Virtual reality and augmented reality for industrial automation

ABSTRACT

An industrial visualization system generates and delivers virtual reality (VR) and augmented reality (AR) presentations of industrial facilities to wearable appliances to facilitate remote or enhanced interaction with automation systems within the facility. VR presentations can comprise three-dimensional (3D) holographic views of a plant facility or a location within a plant facility. The system can selectively render a scaled down view that renders the facility as a 3D scale model, or as a first-person view that renders the facility as a full-scale rendition that simulates the user&#39;s presence on the plant floor. Camera icons rendered in the VR presentation can be selected to switch to a live video stream generated by 360-degree cameras within the plant. The system can also render workflow presentations that guide users through the process of correcting detected maintenance issues.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of, and claims priority to, U.S.patent application Ser. No. 15/718,856, filed on Sep. 28, 2017, andentitled “VIRTUAL REALITY AND AUGMENTED REALITY FOR INDUSTRIALAUTOMATION,” which claims priority to U.S. Provisional Application Ser.No. 62/419,061, filed on Nov. 8, 2016, entitled “VIRTUAL REALITY ANDAUGMENTED REALITY FOR INDUSTRIAL AUTOMATION,” the entirety of which isincorporated herein by reference.

BACKGROUND

The subject matter disclosed herein relates generally to industrialautomation systems, and, more particularly, to visualization ofindustrial data

BRIEF DESCRIPTION

The following presents a simplified summary in order to provide a basicunderstanding of some aspects described herein. This summary is not anextensive overview nor is intended to identify key/critical elements orto delineate the scope of the various aspects described herein. Its solepurpose is to present some concepts in a simplified form as a prelude tothe more detailed description that is presented later.

In one or more embodiments, a system is provided, comprising a deviceinterface component configured to receive industrial data fromindustrial devices associated with an industrial facility; a videoprocessing component configured to receive and store video data fromvideo capture devices installed at respective locations of theindustrial facility; a rendering component configured to generateaugmented reality presentation data that renders a augmented realitypresentation of the industrial facility on a wearable appliance, whereinthe augmented reality presentation comprises camera icons rendered atlocations within the augmented reality presentation corresponding to thelocations of the industrial facility at which the video capture devicesare installed; and a client interface component configured to send theaugmented reality presentation data to the wearable appliance, whereinthe rendering component is further configured to, in response to receiptof camera selection data from the wearable appliance indicating that acamera icon of the camera icons has been selected via user input,transition the augmented reality presentation from a virtual view of theindustrial facility to a video presentation that renders, on thewearable device, a subset of the video data received from a videocapture device, of the video capture devices, corresponding to thecamera icon.

Also, in one or more embodiments a method is provided, comprisingreceiving, by a system comprising a processor, industrial data generatedby industrial devices of an industrial facility; receiving, by thesystem, video data generated by video capture devices installed atrespective locations within the industrial facility; rendering, by thesystem on a wearable appliance based on the industrial data, anaugmented reality representation of the industrial facility, wherein therendering comprises rendering camera icons at locations on the augmentedreality representation corresponding to the locations within theindustrial facility; in response to receipt of camera selection datafrom the wearable appliance identifying a camera icon, of the cameraicons, that has been selected via user input, transitioning, by thesystem, the augmented reality representation from a virtual view of theindustrial facility to a video presentation; and streaming, by thesystem to the wearable device as the video presentation, a subset of thevideo data received from a video capture device, of the video capturedevices, corresponding to the camera icon.

Also, according to one or more embodiments, a non-transitorycomputer-readable medium is provided having stored thereon instructionsthat, in response to execution, cause a system to perform operations,the operations, comprising receiving industrial data generated byindustrial devices of an industrial facility; receiving video datagenerated by video capture devices installed at respective locationswithin the industrial facility; rendering, on a wearable appliance basedon the industrial data, a augmented reality presentation of theindustrial facility; rendering camera icons at locations within theaugmented reality presentation corresponding to the locations within theindustrial facility; and in response to receiving camera selection datagenerated by the wearable appliance identifying a camera icon, of thecamera icons, that has been selected via user input, transitioning theaugmented reality presentation from a virtual view of the industrialfacility to a video presentation, wherein the video presentationcomprises a rendering of a subset of the video data generated a videocapture device, of the video capture devices, represented by the cameraicon.

To the accomplishment of the foregoing and related ends, certainillustrative aspects are described herein in connection with thefollowing description and the annexed drawings. These aspects areindicative of various ways which can be practiced, all of which areintended to be covered herein. Other advantages and novel features maybecome apparent from the following detailed description when consideredin conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example industrial control environment.

FIG. 2 is a conceptual diagram illustrating presentation of augmented orvirtual reality presentations to a wearable appliance or computingdevice worn by a user.

FIG. 3 is a block diagram of an example virtual and augmented realitypresentation system.

FIG. 4 is a block diagram of an example wearable appliance.

FIG. 5 is a block diagram of a generalized example architectureincluding a VR/AR presentation system that serves as a content providefor augmented and virtual reality presentations of an industrialfacility.

FIG. 6A is a block diagram illustrating components and data flows of aVR/AR presentation system.

FIG. 6B is a block diagram illustrating components and data flows of aVR/AR presentation system in which the system comprises separate contentprovider and rendering systems.

FIG. 7 is a diagram illustrating data inputs leveraged by a VR/ARpresentation system to generate virtual and augmented realitypresentations.

FIG. 8 is a rendition of an example VR/AR reality presentation depictingan industrial area, which can be generated by a rendering component of aVR/AR presentation system.

FIG. 9 is another rendition of the example VR/AR reality presentationdepicting the industrial area.

FIG. 10 is another rendition of the example VR/AR reality presentationdepicting the industrial area.

FIG. 11 is an example first-person view of a production area generatedby a rendering component based on plant models and a user's currentorientation.

FIG. 12 is a diagram illustrating data flows between a VR/ARpresentation system, a wearable appliance, and an industrial controller.

FIG. 13 is a diagram illustrating an example data flow that facilitatescommunication between a wearable appliance and one or more industrialdevices for generation of virtual reality or augmented realitypresentations.

FIGS. 14A and 14B are diagrams illustrating an example configurationthat incorporates video information into VR presentations generated bythe VR/AR presentation system.

FIG. 15 is a perspective of an example VR presentation including acamera icon.

FIG. 16 is a diagram illustrating example data flows between a VR/ARpresentation system, industrial equipment, and a wearable appliance fordelivery of interactive workflow presentations.

FIG. 17 is a diagram illustrating real-time workflow feedback beingdelivered to a wearable appliance as a user is correcting a downtimeissue.

FIG. 18 is a generalized block diagram illustrating interactions betweena control program being tested and a VR/AR model of an industrialfacility.

FIG. 19A is a flowchart of a first part of an example methodology forcreating and rendering VR/AR presentations of industrial facilities withthe option to transition to live video feeds.

FIG. 19B is a flowchart of a second part of the example methodology forcreating and rendering VR/AR presentations of industrial facilities withthe option to transition to live video feeds.

FIG. 19C is a flowchart of a second part of the example methodology forcreating and rendering VR/AR presentations of industrial facilities withthe option to transition to live video feeds.

FIG. 20A is a flowchart of a first part of an example methodology forcontrolling a VR/AR presentation of an industrial facility.

FIG. 20B is a flowchart of a second part of the example methodology forcontrolling a VR/AR presentation of an industrial facility.

FIG. 21 is a flowchart of an example methodology for interfacing with anindustrial automation system via a virtual control panel using awearable appliance.

FIG. 22A is a flowchart of a first part of an example methodology forgenerating augmented reality presentations to guide plant personnelthrough the process of addressing a detected maintenance issue.

FIG. 22B is a flowchart of a second part of the example methodology forgenerating augmented reality presentations to guide plant personnelthrough the process of addressing a detected maintenance issue.

FIG. 23 is an example computing environment.

FIG. 24 is an example networking environment.

DETAILED DESCRIPTION

The subject disclosure is now described with reference to the drawings,wherein like reference numerals are used to refer to like elementsthroughout. In the following description, for purposes of explanation,numerous specific details are set forth in order to provide a thoroughunderstanding thereof. It may be evident, however, that the subjectdisclosure can be practiced without these specific details. In otherinstances, well-known structures and devices are shown in block diagramform in order to facilitate a description thereof.

As used in this application, the terms “component,” “system,”“platform,” “layer,” “controller,” “terminal,” “station,” “node,”“interface” are intended to refer to a computer-related entity or anentity related to, or that is part of, an operational apparatus with oneor more specific functionalities, wherein such entities can be eitherhardware, a combination of hardware and software, software, or softwarein execution. For example, a component can be, but is not limited tobeing, a process running on a processor, a processor, a hard disk drive,multiple storage drives (of optical or magnetic storage medium)including affixed (e.g., screwed or bolted) or removable affixedsolid-state storage drives; an object; an executable; a thread ofexecution; a computer-executable program, and/or a computer. By way ofillustration, both an application running on a server and the server canbe a component. One or more components can reside within a processand/or thread of execution, and a component can be localized on onecomputer and/or distributed between two or more computers. Also,components as described herein can execute from various computerreadable storage media having various data structures stored thereon.The components may communicate via local and/or remote processes such asin accordance with a signal having one or more data packets (e.g., datafrom one component interacting with another component in a local system,distributed system, and/or across a network such as the Internet withother systems via the signal). As another example, a component can be anapparatus with specific functionality provided by mechanical partsoperated by electric or electronic circuitry which is operated by asoftware or a firmware application executed by a processor, wherein theprocessor can be internal or external to the apparatus and executes atleast a part of the software or firmware application. As yet anotherexample, a component can be an apparatus that provides specificfunctionality through electronic components without mechanical parts,the electronic components can include a processor therein to executesoftware or firmware that provides at least in part the functionality ofthe electronic components. As further yet another example, interface(s)can include input/output (I/O) components as well as associatedprocessor, application, or Application Programming Interface (API)components. While the foregoing examples are directed to aspects of acomponent, the exemplified aspects or features also apply to a system,platform, interface, layer, controller, terminal, and the like.

As used herein, the terms “to infer” and “inference” refer generally tothe process of reasoning about or inferring states of the system,environment, and/or user from a set of observations as captured viaevents and/or data. Inference can be employed to identify a specificcontext or action, or can generate a probability distribution overstates, for example. The inference can be probabilistic—that is, thecomputation of a probability distribution over states of interest basedon a consideration of data and events. Inference can also refer totechniques employed for composing higher-level events from a set ofevents and/or data. Such inference results in the construction of newevents or actions from a set of observed events and/or stored eventdata, whether or not the events are correlated in close temporalproximity, and whether the events and data come from one or severalevent and data sources.

In addition, the term “or” is intended to mean an inclusive “or” ratherthan an exclusive “or.” That is, unless specified otherwise, or clearfrom the context, the phrase “X employs A or B” is intended to mean anyof the natural inclusive permutations. That is, the phrase “X employs Aor B” is satisfied by any of the following instances: X employs A; Xemploys B; or X employs both A and B. In addition, the articles “a” and“an” as used in this application and the appended claims shouldgenerally be construed to mean “one or more” unless specified otherwiseor clear from the context to be directed to a singular form.

Furthermore, the term “set” as employed herein excludes the empty set;e.g., the set with no elements therein. Thus, a “set” in the subjectdisclosure includes one or more elements or entities. As anillustration, a set of controllers includes one or more controllers; aset of data resources includes one or more data resources; etc.Likewise, the term “group” as utilized herein refers to a collection ofone or more entities; e.g., a group of nodes refers to one or morenodes.

Various aspects or features will be presented in terms of systems thatmay include a number of devices, components, modules, and the like. Itis to be understood and appreciated that the various systems may includeadditional devices, components, modules, etc. and/or may not include allof the devices, components, modules etc. discussed in connection withthe figures. A combination of these approaches also can be used.

Industrial controllers and their associated I/O devices are central tothe operation of modern automation systems. These controllers interactwith field devices on the plant floor to control automated processesrelating to such objectives as product manufacture, material handling,batch processing, supervisory control, and other such applications.Industrial controllers store and execute user-defined control programsto effect decision-making in connection with the controlled process.Such programs can include, but are not limited to, ladder logic,sequential function charts, function block diagrams, structured text, orother such platforms.

FIG. 1 is a block diagram of an example industrial control environment100. In this example, a number of industrial controllers 118 aredeployed throughout an industrial plant environment to monitor andcontrol respective industrial systems or processes relating to productmanufacture, machining, motion control, batch processing, materialhandling, or other such industrial functions. Industrial controllers 118typically execute respective control programs to facilitate monitoringand control of industrial devices 120 making up the controlledindustrial systems. One or more industrial controllers 118 may alsocomprise a soft controller executed on a personal computer or otherhardware platform, or a hybrid device that combines controllerfunctionality with other functions (e.g., visualization). The controlprograms executed by industrial controllers 118 can comprise anyconceivable type of code used to process input signals read from theindustrial devices 120 and to control output signals generated by theindustrial controllers, including but not limited to ladder logic,sequential function charts, function block diagrams, or structured text.

Industrial devices 120 may include both input devices that provide datarelating to the controlled industrial systems to the industrialcontrollers 118, and output devices that respond to control signalsgenerated by the industrial controllers 118 to control aspects of theindustrial systems. Example input devices can include telemetry devices(e.g., temperature sensors, flow meters, level sensors, pressuresensors, etc.), manual operator control devices (e.g., push buttons,selector switches, etc.), safety monitoring devices (e.g., safety mats,safety pull cords, light curtains, etc.), and other such devices. Outputdevices may include motor drives, pneumatic actuators, signalingdevices, robot control inputs, valves, and the like.

Industrial controllers 118 may communicatively interface with industrialdevices 120 over hardwired or networked connections. For example,industrial controllers 118 can be equipped with native hardwired inputsand outputs that communicate with the industrial devices 120 to effectcontrol of the devices. The native controller I/O can include digitalI/O that transmits and receives discrete voltage signals to and from thefield devices, or analog I/O that transmits and receives analog voltageor current signals to and from the devices. The controller I/O cancommunicate with a controller's processor over a backplane such that thedigital and analog signals can be read into and controlled by thecontrol programs. Industrial controllers 118 can also communicate withindustrial devices 120 over a network using, for example, acommunication module or an integrated networking port. Exemplarynetworks can include the Internet, intranets, Common Industrial Protocol(CIP), Ethernet, DeviceNet, ControlNet, Data Highway and Data HighwayPlus (DH/DH+), Remote I/O, Fieldbus, Modbus, Profibus, wirelessnetworks, serial protocols, and the like. The industrial controllers 118can also store persisted data values that can be referenced by thecontrol program and used for control decisions, including but notlimited to measured or calculated values representing operational statesof a controlled machine or process (e.g., tank levels, positions,alarms, etc.) or captured time series data that is collected duringoperation of the automation system (e.g., status information formultiple points in time, diagnostic occurrences, etc.).

Industrial automation systems often include one or more human-machineinterfaces (HMIs) 114 that allow plant personnel to view telemetry andstatus data associated with the automation systems, and to control someaspects of system operation. HMIs 114 may communicate with one or moreof the industrial controllers 118 over a plant network 116, and exchangedata with the industrial controllers to facilitate visualization ofinformation relating to the controlled industrial processes on one ormore pre-developed operator interface screens. HMIs 114 can also beconfigured to allow operators to submit data to specified data tags ormemory addresses of the industrial controllers 118, thereby providing ameans for operators to issue commands to the controlled systems (e.g.,cycle start commands, device actuation commands, etc.), to modifysetpoint values, etc. HMIs 114 can generate one or more display screensthrough which the operator interacts with the industrial controllers118, and thereby with the controlled processes and/or systems. Exampledisplay screens can visualize present states of industrial systems ortheir associated devices using graphical representations of theprocesses that display metered or calculated values, employ color orposition animations based on state, render alarm notifications, oremploy other such techniques for presenting relevant data to theoperator. Data presented in this manner is read from industrialcontrollers 118 by HMIs 114 and presented on one or more of the displayscreens according to display formats chosen by the HMI developer.

Typically, in order to view information relating to the industrialprocesses carried out by the machines and devices that make upindustrial control environment 100, users must either rely on thepre-developed interface display screens executing on HMIs 114 (see user122), or directly connect to the devices using a portable computer inorder to view control programming and device configurations (see user124). While these data visualization systems allow a user to viewrelevant data values and alarms associated with the various machines anddevices, the localized nature of these systems requires the user to bephysically near an HMI terminal or industrial controller in order toview operational and status data for a given industrial system ormachine. Moreover, HMI displays and controller programming tools providelittle in the way of trouble-shooting guidance or analysis in the eventof a machine fault or other performance issue. Typically, the manner ofpresenting machine and device data via HMI screens or controllerprogramming tools requires the user to visually correlate the datapresented on the screens with the user's own direct view of the relevantmachines or devices.

When diagnosing problems, maintenance personnel are often required tosearch several of these sources of information individually, usingseveral different software packages specific to the respective datasources being searched. Moreover, searching for information pertainingto a particular device or machine often requires an extensive knowledgeof the overall industrial system in order to locate the data source tobe searched (e.g., in order to locate the appropriate industrialcontroller or HMI terminal), as well as to identify the relevantoperator screens and control program routines. Individually searchingeach of these data sources in connection with solving a system downtimeissue or other problem can delay correction of maintenance issues,resulting in lost revenue and scheduling problems. Also, if an operatoror maintenance person is not near an information source—such as an HMIterminal—at the time an operational or maintenance issue occurs, theuser may not be notified of the issue in a timely fashion.

To address these and other issues, one or more embodiments of thepresent disclosure provide a system that generates and deliversaugmented reality (AR) or virtual reality (VR) presentations (referredto collectively herein as “VR/AR presentations”) to a user via awearable computer or other client device. VR/AR presentations generatedby the system can comprise three-dimensional (3D) holographic views of aplant facility or a location within a plant facility (e.g., a work area,a production line, etc.). The holographic views can be delivered to awearable visualization computer, which renders the 3D view as a functionof the user's current location and/or orientation. The system can rendera scaled down view of the factory floor area, which affords the user anexternal overview of the area. This external view can include real-timeavatars representing human operators, superimposed production statisticsand status data, and other information. In accordance with userselection input, the system can switch from this external view to aninternal view that renders a realistic presentation of the factory floorarea from the point of view of a person standing within the environment.This internal view can include superimposed operational and status dataplaced on or near representations of the relevant industrial devices orcontrol panels.

The presentation system 302 can also be configured to work inconjunction with video capture devices (e.g., 360-degree cameras,webcams, swivel-based IP cameras, etc.) installed at one or morelocations within the plant environment. In an example implementation inwhich 360-degree cameras are integrated with the system, presentationsystem 302 can deliver, on request, a live 360-degree video feed of theplant floor to the user's wearable appliance 206. For example, theexternal view described above can include camera icons representing the360-degree cameras that are capable of providing a live video feed. Inresponse to a gesture performed by the user and recognizable by theuser's wearable visualization device (or in response to a recognizablenatural language verbal command), the presentation system can switchfrom the external view to a 360-degree live video feed delivered to theuser's wearable device. The video's angle of view can be changed inaccordance with the user's current direction of view or headorientation, providing a realistic live view of the plant floor to auser at a remote location. In some embodiments, the presentation systemcan enhance or augment this live view with superimposed operational orstatus data positioned on or near the view representations of relevantmachines or devices, thereby yielding an augmented reality view of theenvironment.

For users that are physically located on the plant floor, the VR/ARpresentation system can provide automation system data, notifications,and proactive guidance to the user via modification of the user's viewof his or her immediate surroundings. Such modifications can include,for example, superimposing data values or indicators on a user's view ofa machine or automation system through the user's wearable computer (orother client device capable of rendering a substantially real-time viewof the machine or system). The system can customize presentation of thisinformation based on the user's role, location, line of sight, type ofwearable device, and/or other contextual information.

In general, the presentation system can obtain “real world” images of anindustrial automation device having at least one object via a wearableappliance having at least one image sensory input. The systemcomplements the real-world images on the appliance with virtual oraugmented reality images, data, and the like that are associated with atleast one identified object of the industrial automation system. Thereal world industrial automation device or the at least one identifiedobject can be displayed on the appliance together with avirtual/augmented attribute display of the real world industrialautomation device or the at least one object. The virtual or augmentedreality presentations can include, but are not limited to, revisioninformation, topology information, controls, firmware, connections,problems, alarms, training, human machine interface, location ofcontroller/equipment, maps, manuals, instructions, line diagrams, ladderprograms, locations, avatars, filtered views, cameras, x-ray views,removable views, troubleshooting, how-to's, error proofing, safetyrobots, customer information, equipment information, filters, line ofsight filters, knowledge sharing portals, work flows, view/grab HMI's,line of sight (including distant line of sight), super power line ofsight, authentication, privilege control, and asset tracking.

FIG. 2 is a conceptual diagram illustrating presentation of augmented orvirtual reality presentations 204 to a wearable appliance 206 orcomputing device worn by a user. The wearable appliance 206 can compriseany suitable wearable or portable computing device or appliance capableof rendering a virtual reality or augmented reality presentation thatsubstantially surrounds the user's field of view. As will be describedin more detail herein, user interactions with the AR or VR presentations204 are facilitated by data exchange between a user's wearable appliance206 and a VR/AR presentation system that acts as a content provider.However, some embodiments of wearable appliance 206 can also beconfigured to establish direct communication channels with an industrialdevice in order to send control instructions to such devices. To thisend, one or more embodiments of wearable appliance 206 can includecommunication stacks that establish connectivity between industrialsystems residing on an industrial network—such as a common industrialprotocol (CIP) network, a DeviceNet network, a ControlNet network, anEthernet IP network, or other networks—and the wearable appliance 206.The wearable appliance 206 can comprise any suitable wearable orportable computing device or appliance capable of rendering a virtualreality or augmented reality presentation.

In response to various conditions, such as the user's determined role,location, line of sight, or other information, the system can generateand deliver augmented or virtual reality presentations to the user'swearable appliance 206. Data used to populate the presentations 204 canbe obtained by the VR/AR presentation system from the relevantindustrial devices and delivered as part of the VR/AR presentations 204.In some scenarios, wearable appliance 206 can also obtain at least aportion of the industrial data directly from the industrial devices viathe industrial network by virtue of a communication stack thatinterfaces the wearable appliance 206 to the various devices on thenetwork. Such devices can include individual devices such ascontrollers, human machine interface (HMI) devices, and motor drives, aswell as collections of devices that are housed in a control panel orthat make up a controlled industrial machine or system. The VR/ARpresentation system can customize the presentations 204 based on auser's current context, line of sight, type of client device being usedby the user (e.g., wearable computer, handheld device, etc.), and/orother relevant information, such that customized augmented reality orvirtual reality presentations can be generated based on relevant subsetsof data available on the industrial network.

In an example scenario, as a user is viewing an automation system,machine, or industrial device through a wearable computer (or as asubstantially real-time video image rendered on the user's clientdevice), the VR/AR presentation system can monitor the wearable computerto determine the user's location relative to the automation system, theuser's current line of sight or field of view, and/or other contextualinformation indicative of the user's relationship to the automationsystem. Based on the determined identity of the automation systemcurrently being viewed by the user, the VR/AR presentation system candetermine current status information for devices and/or machines thatmake up the automation system, or for a process being carried out by theautomation system. The VR/AR presentation system can then generateaugmented reality or virtual reality presentations and deliver thesepresentations to the user's wearable appliance; e.g., as graphical ortext-based indicators overlaid on the user's field of view, such thateach indicator is positioned near the machine or device to which theindicator pertains. For example, if the user's current view encompassesa real or virtualized motor-driven conveyor and a motor drive thatcontrols the motor, the presentation system may superimpose a currentoperating status of the motor drive (e.g., a current speed, a faultcondition, an operating mode, etc.) near the image or view of the motordrive as perceived by the user. If the user is currently viewing adie-cast furnace, the presentation system may superimpose a currentfurnace temperature near the view of the furnace.

In yet another example, a monitoring component of the VR/AR presentationsystem can identify a maintenance issue based on analysis ofsubstantially real-time system data generated by the automation system.In response to detecting such a maintenance issue, the presentationsystem can deliver a notification to a wearable appliance or otherclient device associated with a qualified plant technician. To assistthe selected user in locating the source of the detected problem, theVR/AR presentation system can superimpose graphics on the user's view ofhis or her environment that guide the user to the source of the issue.These graphics can include, for example, arrows or other indicators thatguide the user to the affected machine or device, as well as indicatorsthat direct the user's focus of attention to specific areas orcomponents of an automation system, machine, or industrial devicerequiring attention.

FIG. 3 is a block diagram of an example virtual and augmented realitypresentation system 302 (also referred to herein as VR/AR presentationsystem) according to one or more embodiments of this disclosure. Aspectsof the systems, apparatuses, or processes explained in this disclosurecan constitute machine-executable components embodied within machine(s),e.g., embodied in one or more computer-readable mediums (or media)associated with one or more machines. Such components, when executed byone or more machines, e.g., computer(s), computing device(s), automationdevice(s), virtual machine(s), etc., can cause the machine(s) to performthe operations described.

VR/AR presentation system 302 can include a client interface component304, an authentication component 306, a rendering component 308, areporting component 310, a video processing component 312, a deviceinterface component 314, a monitoring component 316, one or moreprocessors 320, and memory 322. In various embodiments, one or more ofthe client interface component 304, authentication component 306,rendering component 308, reporting component 310, video processingcomponent 312, device interface component 314, monitoring component 316,the one or more processors 320, and memory 322 can be electricallyand/or communicatively coupled to one another to perform one or more ofthe functions of the VR/AR presentation system 302. In some embodiments,components 304, 306, 308, 310, 312, 314, and 316 can comprise softwareinstructions stored on memory 322 and executed by processor(s) 320.VR/AR presentation system 302 may also interact with other hardwareand/or software components not depicted in FIG. 3. For example,processor(s) 320 may interact with one or more external user interfacedevices, such as a keyboard, a mouse, a display monitor, a touchscreen,or other such interface devices.

Client interface component 304 can be configured to exchange informationbetween the VR/AR presentation system 302 and a wearable appliance orother client device having authorization to access the system. Forexample, the client interface component 304 can receive contextualinformation about a user based on a monitoring of the user's wearableappliance or other client device, device or machine identity information(e.g., information obtained by the wearable appliance from optical codesassociated with the device or machine), requests from the wearableappliance to add or remove information from the presentation, commandsfrom the wearable appliance to transition the presentation to a livevideo feed sourced by a selected video camera, requests from thewearable appliance to invoke a virtual control panel or other virtual oraugmented reality presentation, etc. Client interface component 304 canalso deliver augmented reality, virtual reality, or mixed realitypresentations to the wearable appliance.

Authentication component 306 can be configured to confirm authorizationof a user to receive and interact with a virtual control panel or othervirtual or augmented reality presentation. For example, authenticationcomponent 306 can be configured to cross-reference user identificationinformation received from a wearable appliance with control privilegeinformation defined for the identified user. Authentication component306 may also determine a defined role associated with the useridentification information, and grant a level of control privilegecommensurate with the user's role. Levels of control privilegecontrolled by authentication component 306 can include, for example,view-only privileges, full control privileges, limited controlprivileges whereby a selected subset of virtual control panel functionsmay be interfaced by the user, or other such access levels.

Rendering component 308 can be configured to retrieve a suitable virtualreality or augmented reality presentation for rendering on a user'swearable appliance, and modify or enhance the presentation withreal-time or historical data retrieved from one or more industrialdevices, live or historical video feeds of the plant floor, or otherinformation. In the case of augmented reality presentations delivered tothe user's wearable appliance as the user traverses the plantenvironment, some embodiments of rendering component 308 can generatepresentations based on an identity of an industrial device, automationsystem, control cabinet, or machine received from the wearableappliance, such that available information about devices, machines, orcontrol cabinets within the user's line of sight is displayed on theappliance. The rendering component 308 can also select the VR/ARpresentation in accordance with the user's control privileges(determined by the authentication component 306). The selectedpresentation can then be sent to the wearable appliance the clientinterface component 304.

Reporting component 310 can be configured to generate report data basedon computations performed on subsets of collected industrial data, andpresent the report data in a suitable format on a VR/AR presentation viathe wearable appliance. For example, reporting component 310 can beconfigured to calculate operating statistics for a device, work cell,machine, or production area based on data collected from industrialdevices on the plant floor. The rendering component 308 can then renderthese statistics on an augmented or virtual reality presentation. Videoprocessing component 312 can be configured to process and store videostream data from one or more cameras mounted on the plant floor, suchthat the video data from each camera is tagged with identificationinformation indicating the location recorded by the video data.Rendering component 308 can, in response gesture or verbal inputreceived from a user's wearable appliance, transition a VR/ARpresentation to a live or historical video feed sourced by the storedvideo data.

Device interface component 314 can be configured to exchange informationbetween the VR/AR presentation system 302 and one or more on-premiseindustrial devices (e.g., industrial controllers, telemetry devices,motor drives, quality check systems, industrial safety systems, etc.),cameras, or data collection devices (e.g., industrial data historians),located at one or more industrial plant facilities. In some embodiments,device interface component 314 can exchange data with the on-premisedevices via the plant networks on which the devices reside. In someembodiments, device interface component 314 can also receive some or allof the plant floor data via a public network such as the Internet. Thedevice interface component 314 can directly access the data generated bythese on-premise industrial devices and systems via the one or morepublic and/or private networks in some embodiments. Alternatively,device interface component 314 can access the data on these on-premisedevices via a proxy or gateway device that aggregates the data frommultiple industrial devices for migration to the cloud platform via thedevice interface component. The data received by the device interfacecomponent 314.

Monitoring component 316 can be configured to monitor selected subsetsof data collected by device interface component 314 according to definedmonitoring rules, and to deliver notifications and/or workflowrecommendations in response to detecting a maintenance or performanceissue based on a result of the monitoring. Monitoring component 316 canwork in conjunction with rendering component 308 to deliver suitablenotifications and workflows to wearable appliances associated withappropriate plant personnel, such that the workflows are presented aspart of an augmented reality presentation to guide personnel through theprocess of enacting an appropriate countermeasure to the detected issue.In addition to defining the conditions that define an issue requiringnotification, the monitoring rules can also define which employees areto be notified in response to each type of detected performance ormaintenance issue.

The one or more processors 320 can perform one or more of the functionsdescribed herein with reference to the systems and/or methods disclosed.Memory 322 can be a computer-readable storage medium storingcomputer-executable instructions and/or information for performing thefunctions described herein with reference to the systems and/or methodsdisclosed.

FIG. 4 is a block diagram of an example wearable appliance 206 accordingto one or more embodiments of this disclosure. Wearable appliance 206can include a system interface component 404, a device communicationcomponent 406, a visualization component 408, a location and orientationcomponent 410, one or more processors 420, and memory 422. In variousembodiments, one or more of the system interface component 404, devicecommunication component 406, visualization component 408, location andorientation component 410, the one or more processors 420, and memory422 can be electrically and/or communicatively coupled to one another toperform one or more of the functions of the wearable appliance 206. Insome embodiments, components 404, 406, 408, and 410 can comprisesoftware instructions stored on memory 422 and executed by processor(s)420. Wearable appliance 206 may also interact with other hardware and/orsoftware components not depicted in FIG. 4. For example, processor(s)420 may interact with one or more external user interface devices, suchas a keyboard, a mouse, a display monitor, a touchscreen, or other suchinterface devices.

System interface component 404 can be configured to exchange data overwireless communication channels with VR/AR presentation system 302.Device communication component 406 can be configured to exchange databetween the wearable appliance 206 and industrial devices via anindustrial network on which the devices reside. In an exampleimplementation for use with CIP networks, the device communicationcomponent 406 can support CIP protocol carried by EtherNet/IP. However,embodiments described herein are not limited to these protocols.

Visualization component 408 can be configured to render the virtualreality, augmented reality, mixed reality, or video presentationsdelivered to the wearable appliance 206 by VR/AR presentation system302. Example augmented reality presentations can include graphicaloverlays that are superimposed over a user's field of view of his or hersurroundings via a wearable appliance. These graphical overlays caninclude, but are not limited to, operational or status data indicators(both alphanumerical and icon-based indicators) for an industrial systemor device within the user's field of view, indicators that direct a userto a location of an industrial system or device within a plantenvironment, guidance indicators for assisting a user in diagnosing andaddressing an identified problem with an industrial system or device, orother such overlays. Example VR/AR presentations can include bothexternal scaled down views of a factory floor area as well asvirtualized first-person views of the plant floor. Visualizationcomponent 408 can also render, under the instruction of VR/ARpresentation system 302, live or pre-recorded video feeds received from360-degree cameras (or other types of video or audio capture devices)mounted at selected areas of the plant floor.

Location and orientation component 410 can be configured to determine alocation and an orientation of the wearable appliance 206. Thisinformation can be sent to the VR/AR presentation system 302 by systeminterface component 404 so that human operators can be tracked andrendered within a VR presentation, and so that the VR/AR presentationrendered by visualization component 408 reflects the user's currentlocation and/or orientation.

The one or more processors 420 can perform one or more of the functionsdescribed herein with reference to the systems and/or methods disclosed.Memory 422 can be a computer-readable storage medium storingcomputer-executable instructions and/or information for performing thefunctions described herein with reference to the systems and/or methodsdisclosed.

FIG. 5 is a block diagram of a generalized example architectureincluding a VR/AR presentation system 302 that renders augmented andvirtual reality presentations of an industrial facility. The exampleindustrial environment depicted in FIG. 5 includes one or moreindustrial controllers 504, HMIs 506, motor drives 518, industrialsafety systems 520, databases 508 (e.g., data historians, employeedatabases, inventory databases, etc.), and device documentationrepositories 510. The industrial environment may also include othersources of industrial data not depicted in FIG. 5, including but notlimited to quality systems (e.g., vision systems or other qualifyverification systems), telemetry devices, presence sensors (e.g., photodetectors, optical scanners, proximity switches, etc.), video cameras,and other devices or sub-systems. In an example environment, theseindustrial devices and systems can reside on a plant (operationaltechnology) network 116. In some scenarios, the industrial devices maybe distributed across multiple plant networks 116 within the plantfacility. The industrial environment may also include devices andsystems residing on an office (information technology) network 108,including but not limited to manufacturing execution systems (MES) 526,enterprise resource planning (ERP) systems 528, business intelligencesystems, business-level analytic systems, or other such assets. One orboth of office network 108 or plant network 116 may also have access toexternal networks 514 such as the Internet (e.g., via firewall device516).

VR/AR presentation system 302—which resides on plant network 116 in theexample architecture depicted in FIG. 5, but which may also reside onoffice network 108, on an external network, on a web server, or on acloud platform as a cloud-based service provider—collects data from thediverse set of industrial devices via network 116. In someconfigurations, the presentation system 302 can also collect selecteditems of plant data from one or more devices or systems on officenetwork 108, including but not limited to the MES system 526, ERP system528, business intelligence systems, or other such assets. Presentationsystem 302 formats the data for rendering in virtual and augmentedreality presentations. One or more plant models 524 stored on thepresentation system 302 can define three-dimensional views of areaswithin an industrial facility (e.g., production lines or work areas),and presentation system 302 can generate three-dimensional virtual oraugmented reality presentations of the areas—including machines, controlcabinets, conveyors, industrial devices, etc.—based on the plant models524. The presentation system 302 can also superimpose selected subsetsof the collected industrial data on the virtual or augmented realitypresentations on or near graphical representations of the industrialasset (e.g., machine, control cabinet, industrial controller, etc.) towhich the data relates. Other interactive features of the virtual andaugmented reality presentations will be described in more detail herein.

The virtual and augmented reality presentations can also be customizedin accordance with a defined role of the wearer of appliance 206, asspecified in user profiles 522 defined for each user of the system.Example user roles that can determine how VR and AR data is presented toa user can include, but are not limited to, line operators, maintenancepersonnel, plant managers, plant engineers, or other roles.

Presentation system 302 can deliver these presentations to a wearableappliance 206 worn by a user, who may be at the plant facility or at aremote location relative to the facility. In the case of remote accessfrom outside the facility, presentation system 302 can be made securelyaccessible by authorized wearable appliances 206 via an outside networksuch as the Internet. In some embodiments, presentation system 302 canbe implemented on a web server, allowing wearable appliance 206 toinvoke VR/AR presentations via an Internet connection. The presentationsystem 302 may also be implemented on a networked local serveraccessible by the wearable appliance 206 via a wireless networkconnection. In yet another scenario, presentation system 302 may beimplemented on a cloud platform, where the search system executes as acloud-based service.

FIG. 6A is a block diagram illustrating components of the VR/ARpresentation system 302 in more detail. VR/AR presentation system 302includes a device interface component 314 that collects live andhistorical industrial data from industrial devices and systems 608distributed across an industrial environment. In some embodiments,device interface component 314 can be configured to retrieve selecteddata items from the industrial devices and systems 608 via networks 116or 108 in accordance with defined monitoring rules that specify the dataitems to be collected. The data items to be collected can be defined interms of data tag names that identify data tags on industrialcontrollers, HMIs, data historians, or other industrial devices; nameand location information for data files to be collected (e.g., workorder data files, device documentation files, inventory files, etc.), orother such data identifiers. The collected data items can includetelemetry and status information relating to operation of the devices ortheir associated industrial automation systems, as well as configurationinformation for the industrial devices (e.g., motor drive parameters,industrial controller communication settings, I/O modules installed oneach industrial controller, etc.). From the office network 108 orexternal networks 514, the collected data can include, but is notlimited to, work management information, production line schedulinginformation, operator work schedule information, product or materialinventory information, etc.

For some industrial devices, the device configuration or programdevelopment application used to configure and/or program the device canalso be used to define which data items on the device are to becollected by the VR/AR presentation system 302. For example, the programdevelopment application used to define data tags on an industrialcontroller—as well as to program the controller and configure thecontroller's I/O and communication settings—can include an option toflag data tags defined on the controller for collection and rendering bythe VR/AR presentation system 302. In such embodiments, the programdevelopment application may be integrated with a virtual/augmentedreality configuration tool, so that both the controller and aspects ofthe controller's VR or AR visualization can be configured together usingthe same configuration tool. For example, for a given data tag definedon the industrial controller, the program development application canallow the user to set the tag to be a value that is to be collected bythe AR/VR presentation system, as well as to define any associations thetag may have outside the scope of the controller (e.g., by identifyingany production areas, machines, industrial processes, or automationsystems the data tag is associated with). The user may also define thevisualization privileges associated with the tag via the programdevelopment application, which can be used by rendering component 308 todetermine which user roles are permitted to view data associated withthe data tag. Based on such configuration information, renderingcomponent 308 can render selected items of data defined on theindustrial controller (or other industrial devices) in association withthe virtualized production area, machines, processes, or systems withwhich the data tag has been assigned, and in accordance with the definedrole-based visualization privileges.

In some embodiments, the device interface component 314 can also beconfigured to discover data items residing on industrial devicesdistributed across the environment. In some embodiments, deviceinterface component 314 can discover available data items by deployingdiscovery agents on network 116 and/or 108. These agents—which can beprograms or bots—can traverse networks 116 and/or 108 and identifydevices in use throughout the plant, as well as the data items or tags,applications, and configuration information associated with thosedevices. Since a given industrial environment typically comprises aheterogeneous collection of devices of different types and vendors, andthe data made available by these devices may comprise many differentdata types (e.g., controller tags, HMI tags, alarms, notifications,events, etc.), some embodiments of device interface component 314 canmanage and deploy device-specific or platform-specific agents configuredto extract and analyze information from specific types of devices ordata platforms (e.g., controllers, HMIs, etc.). Some device-specificagents can be configured to locate application project files stored onparticular device types (e.g., configuration and/or program files on anindustrial controller, screen configuration files on an HMI, etc.), andextract relevant information about the devices based on analysis of datacontained in these project files. By leveraging device-specific andplatform-specific agents, embodiments of device interface component 314can discover and retrieve data conforming to many different formats andplatforms.

In order to unify this disparate heterogeneous data under a commonplatform for collective searching, device interface component 314 (orthe device-specific agents) can transform the collected data to a formatunderstandable by the rendering component 308 to yield normalized plantdata 610.

In some embodiments, device interface component 314 can also discoverand record relationships—both explicit and inferred—between data itemsdiscovered on the industrial devices and systems 608. In suchembodiments, the device interface component 314 may record theserelationships by tagging discovered data items with classification tagsand building a search index based on these classification tags, suchthat related data items share common tags. The classification tags mayidentify, for example, a common machine or automation system with whichthe devices are associated, a production area in which the devicesreside, a control cabinet identifier, or other such classification tags.In some scenarios, these classification tags may be explicitly definedby a system developer such that the device interface component 314determines which predefined tags should be applied to newly discovereddata items. The device interface component 314 may also auto-generateclassification tags for a given data item based on contextualinformation, including but not limited to rung comments associated witha controller tag, learned interdependencies between a newly discovereddata item and a previously discovered data item (e.g., learn that a pumpnamed Pump5 is associated with a tank named Tank1, and therefore tagPump5 as being associated with Tank1, or tag both Tank1 and Pump5according to the larger system in which they operate), or otherdiscovered contextual information. The device interface component 314can define associations between similarly tagged data items regardlessof the platform in which they were discovered. For example, the deviceinterface component 314 can associate common or related data itemsdiscovered, respectively, in an industrial controller, an HMI, a datahistorian, and ERP or MES system, a business intelligence system, etc.

Using some or all of these techniques, device interface component 314can discover and collect operational, status, and configuration datarelating to operation and health of industrial automation systems acrossa facility, as well as higher-level business data from devices on anoffice or IT network. This collected plant data 610 can be stored inmemory associated with the VR/AR presentation system 302 (e.g., memory322) and used by rendering component 308 to populate virtual andaugmented reality presentations with live or historical data.

Although FIG. 6A depicts the components of presentation system 302 asresiding on a common system device, in some embodiments one or more ofthe components of the system 302 can reside on different physicaldevices. FIG. 6B is a diagram of an example implementation in whichpresentation system 302 comprises two separate systems—a renderingsystem 302 a and a content provider 302 b. In this example, contentprovider 302 collects the plant data 610 as described above. Userprofiles 522 and plant models 524 can also reside on the contentprovider in this example implementation. Reporting component 310 can beconfigured to perform any analysis or computations on the plant data inconnection with generated report data (e.g., computed KPIs, efficiencycalculations, etc.). A rendering interface component 612 can provideselected subsets of plant data 610, report data generated by reportingcomponent 310, and data obtained from plant models 524 and user profiles522 to the rendering system 302 a, which renders the VR/AR presentationsand delivers the presentations to wearable appliance 206. Otherdistributions of the components of presentation 302 are also within thescope of one or more embodiments of this disclosure.

Wearable appliance 206 can interface with VR/AR presentation system 302via client interface component 304, which may comprise a wired orwireless network interface, a near-field communication interface, orother such device interface suitable for the particular platform onwhich the presentation system 302 is implemented. In some embodiments,client interface component 304 may be configured to verify anauthorization of the wearable appliance 206 to access the presentationsystem 302 prior to allowing VR/AR presentations to be delivered to thewearable appliance 206. Client interface component 304 may authenticatethe wearable appliance or its owner using password verification,biometric identification (e.g., retinal scan information collected fromthe user by the wearable appliance 206 and submitted to the clientinterface component 304), cross-referencing an identifier of thewearable appliance 206 with a set of known authorized devices, or othersuch verification techniques.

Rendering component 308 is configured to generate virtual and augmentedreality presentation data 604 to wearable appliance 206 for delivery byclient interface component 304. Presentation data 604, when received andexecuted by wearable appliance 206, renders an interactivethree-dimensional virtual reality presentation of an industrial area onthe wearable appliance's display.

The VR/AR presentation is generated based on a combination of diverseinformation received and processed by rendering component 308. FIG. 7 isa diagram illustrating data inputs leveraged by VR/AR presentationsystem 302 to generate VR/AR presentations. As noted above, presentationsystem 302 collects device data 704 from industrial devices or systems702 across the plant environment. Presentation system 302 also maintainsone or more plant models 524 that define a visual representation of thephysical layout of the area represented by a VR presentation. Forexample, a plant model for a given industrial area (e.g., a productionarea, a workcell, an assembly line, etc.) can define graphicalrepresentations of the industrial assets—including machines, conveyors,control cabinets, and/or industrial devices—located within that area, aswell as the physical relationships between these industrial assets. Foreach industrial asset, the plant model can define physical dimensionsand colors for the asset, as well as any animation supported by thegraphical representation (e.g., color change animations, positionanimations that reflect movement of the asset, etc.). The plant models524 also define the physical relationships between the industrialassets, including relative positions and orientations of the assets onthe plant floor, conduit or plumbing that runs between the assets, andother physical definitions.

A rendering engine supported by rendering component 308 is configured togenerate an interactive VR/AR presentation of the industrial area basedon the industrial asset rendering definitions specified in the plantmodels. Rendering component 308 populates this virtual realitypresentation with selected subsets of collected plant data 610 (as wellas production or operational statistics calculated by reportingcomponent 310 based on subsets of the plant data 610), and clientinterface component 304 delivers the resulting aggregate VR/ARpresentation to wearable appliance 206 as VR/AR presentation data 604.Rendering component 308 can generate the presentation such that items ofthe plant data 610 are overlaid on or near graphical representations ofthe industrial assets to which the items of data relate.

FIG. 8 is a partial rendition of an example virtual reality presentationdepicting an industrial area, which can be generated by renderingcomponent 308. It is to be appreciated that, due to the constraintsinherent in presenting virtual reality presentations via two-dimensionaldrawings, the example VR/AR presentations illustrated in FIG. 8 andother figures of this disclosure cannot fully depict the VR/ARpresentations that are rendered on suitable wearable appliances. Ingeneral, the VR/AR presentations rendered by wearable appliances 206provide surrounded virtual renderings that encompass the user's entirefield of view, and transition their line of sight or perspective as theuser's location and orientation change. The partial renditions andassociated descriptions herein seek to convey the virtual realityrenderings and interactions to the degree possible given the limitationsof two-dimensional illustrations.

Rendering component 308 can support both external VR/AR views of theindustrial area from the perspective of a person outside of the area, aswell as first-person views of the area that simulate the user's presencewithin the industrial area by rendering a full-scale view of the area.The view presented in FIG. 8 is an external view of the industrial area,which is generated by the wearable appliance's visualization component408 in accordance with VR/AR presentation data 604 delivered bypresentation system 302. VR/AR presentation system 302 can streamup-to-date VR/AR presentation data 604 to wearable appliance 206 toensure that the view—including the user's angle of perspective and liveindustrial data values—remains current. Industrial assets rendered inthe presentation, including tanks 818, conveyor 812, vats 814, andmachines 816 are rendered by the rendering component 308 in accordancewith rendering instructions defined by one or more of the plant models524.

In the example view depicted in FIG. 8, production statistic icons 802are rendered at a fixed location above the production area. Theseproduction statistic icons 802 display production statistics or keyperformance indicators (KPIs) for the rendered production line. Theproduction statistics can be calculated by reporting component 310 basedon selected subsets of the plant data 610. Example statistics that canbe rendered via production statistic icons 802 include, but are notlimited to, overall equipment effectiveness (OEE), performance orproduction efficiency, percentage of machine availability over time,product quality statistics, cycle times, overall downtime or runtimedurations (e.g., accumulated or per work shift), or other suchstatistics.

Rendering component 308 can also render human icons 808 a and 808 brepresenting human operators present on in the production area.Returning briefly to FIG. 7, in some embodiments the locations andorientations of the human icons 808 a and 808 b within the VR/ARpresentation can be determined based on location and orientation data606 received by VR/AR presentation system 302 from the wearableappliances 206 associated with each user. In this regard, the locationand orientation component 410 of wearable appliance 206 can beconfigured to determine a current geographical location of the appliance206. In some embodiments, location and orientation component 410 canleverage global positioning system (GPS) technology to determine theuser's absolute location, or may be configured to exchange data withpositioning sensors located within the plant facility in order todetermine the user's relative location within the plant. Location andorientation component 410 can also include orientation sensingcomponents that measure the wearable appliance's current orientation interms of the direction of the appliance's line of site, the angle of theappliance relative to horizontal, etc. Other types of sensors oralgorithms can be supported by embodiments of the wearable appliance 206for determining a wearer's current location and orientation, includingbut not limited to inertial measurement units (IMUs) or visual-inertialodometry (VIO). The wearable appliance's system interface component 404can report the location and orientation information generated bylocation and orientation component 410 to the VR/AR presentation system302 as location and orientation data 606.

Location and orientation data 606 is used by VR/AR presentation system302 to both control how human icons 808 a and 808 b are rendered on auser's VR/AR presentation, as well as to control the point of view ofthe VR/AR presentation as a whole. For example, a first user may beviewing a VR presentation of an industrial area (e.g., the externalpresentation depicted in FIG. 8) via the user's wearable appliance 206.Rendering component 308 receives location and orientation data 606generated by the first user's wearable appliance, and renders thepresentation on the wearable appliance in accordance with the firstuser's current location and orientation. In particular, the directionand angle of the viewing perspective of the VR presentation is afunction of the first user's location and orientation.

For example, if the user's wearable appliance 206 is currentlypresenting the view depicted in FIG. 8, and the user moves forward andslightly to the left, the rendering component 308 will transition to theview depicted in FIG. 9. In general, the external view generated byVR/AR presentation system 302 renders the industrial area as a virtualscale model of the area, and allows the user to move around and interactwith the scaled version of the area. As the user moves around, toward,or away from the virtual scaled industrial area, the wearable appliance206 streams updated location and orientation data 606 to thepresentation system 302, which updates the presentation data 604substantially continuously to simulate the effect of walking around ascale model of the production area. Accordingly, moving forward andslightly to the left causes the presentation depicted in FIG. 8 totransition to the presentation depicted in FIG. 9, whereby theperspective is now nearer to and slightly behind the human icons 808 aand 808 b.

FIG. 10 is yet another perspective of the example VR/AR presentation,generated by the rendering component 308 in response to the user'smovement around to the front of the production area being rendered.

As can be seen in FIGS. 8-10, human icons 808 a and 808 b are renderedat locations within the presentation corresponding to actual locationsof human operators within the real production area. To determine theproper location and orientation of human icons 808 a and 808 b,rendering component 308 can leverage location and orientation data 606received from other wearable appliances 206 worn by the human operators.Alternatively, other types of devices carried or worn by the humanoperators and capable of tracking the operators' locations andorientations can also be used to provide VR/AR presentation system 302with operator location and orientation information. Rendering component308 can determine where the human icons 808 a and 808 b are to be placedwithin the virtual production area based on the location informationreceived from the operators' wearable appliances 206, while thedirections in which the human icons 808 a and 808 b face can be based onthe orientation information received from the wearable appliances 206,which indicates the current line of sight for the operators.

In one or more embodiments, rendering component 308 can generate anoperator information icon 804 near each human icon 808 (in FIGS. 8-10,operator information icons 804 a and 804 b are placed near human icons808 a and 808 b, respectively). The wearer of the wearable appliance 206being used to view the VR/AR presentation can select any of theseoperator information icons 804 to invoke information about the humanoperator corresponding to the human icon 808. In some embodiments, thewearable appliance 206 can be configured to recognize selection gesturesperformed by the wearer, and these selection gestures can be used toselect one of the operator information icons 804. As an exampleselection gesture, the user may place his or her hand in front of thewearable appliance 206 near a location corresponding to the desiredoperator information icon 804, and perform a pinching gesture with hisor her thumb and forefinger. The wearable appliance 206 can recognizethe gesture and identify the location within the presentation at whichthe gesture was performed. If the location corresponds to a selectableicon (such as the operator information icon 804), the icon will beselected and an appropriate action performed. Icons can also be selectedusing verbal commands in some embodiments.

Selection of an operator information icon 804 can cause an operatorinformation window to be overlaid near the human icon 808 correspondingto the selected operator information icon 804. Information about eachoperator can be stored on memory 332 associated with VR/AR presentationsystem 302. Since each wearable appliance 206 provides user identitydata 602 in addition to location and orientation data 606, renderingcomponent 308 can cross-reference the received user identity data 602with the stored operator data in response to selection of the operator'sinformation icon 804, and retrieve the subset of operator datacorresponding to the selected operator for presentation in the overlaidoperator information window. Example operator data that can be displayedin response to an operator information icon 804 can include, but is notlimited to, the operator's name and role (e.g., machine operator,engineer, maintenance person, plant manager, etc.), work scheduleinformation, logged work hours (which may also be categorized accordingto hours worked in each of multiple production areas), certificationsheld by the operator (e.g., safety training certifications or othercertifications), experience indications for the operator (e.g., types ofmachines or industrial devices with which the operator has hadoperational or maintenance experience, types of maintenance issues theoperator has addressed in the past), or other such operator-specificinformation.

Some embodiments of VR/AR presentation system 302 can also allow aremote viewer of the VR/AR presentation to open an audio or audio-videochannel to a wearable appliance 206 associated with a user correspondingto a selected one of the human icons 808. For example, in response to anappropriate gesture or verbal command recognizable by the viewing user'swearable appliance 206 as a selection of one of the human icons 808 foraudio or audio-visual communication, rendering component 308 canestablish an audio or audio-visual communication channel between theviewing user's wearable appliance 206 and that of the personcorresponding to the selected human icon 808. Thus, audible informationsuch as speech—or audio-visual information—received by the viewer'swearable appliance 206 will be transmitted to the audio-visual outputcomponents of the selected user's wearable appliance 206. This allowsthe remote user to provide verbal instructions to selected personnel onthe plant floor (e.g., guidance in connection with addressing anoperational or maintenance issue), or to share visual informationbetween the users. Also, for some embodiments in which wearableappliances 206 support generation of haptic feedback to the wearer,VR/AR presentation system 302 can also allow a remote viewer of theVR/AR presentation to initiate, via interaction with a human icon 808, ahaptic signal (e.g., a vibration) directed toward user corresponding tothe human icons 808. This feature can be used to remotely attract theattention of a user while the user is in a noisy environment.

Rendering component 308 can also superimpose asset information icons 810(e.g., 810 a and 810 b) on or near representations of industrial assetsfor which additional information is available. In FIGS. 8-10, assetinformation icon 810 a has been placed near a control cabinet 902, andanother asset information icon 810 b has been placed near machine 904.These asset information icons 810 can be selected using similar gestureor verbal recognition techniques used to select operator informationicons 804. Selection of an asset information icon 810 can causerendering component 308 to render an information window on or near theasset corresponding to the icon 810, and populate this informationwindow with information relevant to the asset. Some or all of therendered asset information can be derived from relevant subsets of plantdata 610, or can be derived from stored data about the asset registeredpreviously with the VR/AR presentation system 302. In the case ofcontrol cabinets, rendered information can include, but is not limitedto, a name of the cabinet, a machine or industrial process controlled bythe control cabinet, identification of devices mounted within thecabinet (e.g., industrial controllers, I/O modules, motor drives,contactors, etc.), statuses of devices mounted within the cabinet, orother such information. In the case of machines, information that can berendered in response to selection of asset information icon 810 b caninclude, but is not limited to, a current operating mode of the machine(e.g. automatic, manual, semi-manual, etc.), a fault status for themachine, an operating speed or production rate, a total number of loggedwork hours (or work hours since a most recent maintenance activity), adate on which maintenance was most recently performed on the machine,information regarding maintenance actions performed on the machine, orother such information.

The VR/AR presentation of the production area also includes a number ofcamera icons 806 (e.g., 806 a-806 d) that allow the user to switch thepresentation to a live or historical video feed, as will be described inmore detail below.

In the example external view illustrated in FIGS. 8-10, renderingcomponent 308 renders a scaled-down version of the production area thatcan be viewed by the user as if the production area were an animated,interactive scale model. This allows the user to virtually walk aroundthe model and view aspects of the production area from an externaloverview perspective. Rendering component 308 also allows the user toswitch to a full-scale rendering of the production area that offers afirst-person perspective of the production area. FIG. 11 is an examplefirst-person view of the production area generated by the renderingcomponent 308 based on the plant model(s) 524 and the user's currentorientation. In some embodiments, rendering component 308 can beconfigured to transition between the external view and the first-personview in response to recognition of a transition gesture or verbalcommand by the user. The first-person view allows the user to virtuallytraverse the rendered representation of the production area from theperspective of a user on the plant floor, even if the user is at aremote location relative to the real plant facility. In someembodiments, while the user is viewing the VR/AR presentation from theexternal perspective, the user can select a target location within theproduction area at which the user wishes to begin the first-personperspective. In response to selecting the target location, the renderingcomponent 308 can smoothly transition from the external perspective tothe first-person perspective by zooming the view toward the targetlocation. As the presentation is zoomed, the virtual production area isscaled upward until the virtual dimensions of the virtual productionarea substantially simulate the user's actual presence on the plantfloor. Smoothly transitioning to the first-person perspective in thismanner can help orient the user within the virtual production area.

While in the first-person view, rendering component 308 can rendersubsets of plant data 610, calculated production or machine statistics,or alphanumeric message as overlaid information placed on or near thevirtual assets (e.g., control cabinets such as control cabinet 1102,machines, control devices, motors drives, valves, tanks, etc.) to whichthe information relates. For example, while the user is viewing avirtual control cabinet (e.g., virtual control cabinet 1102) in thefirst-person view, rendering component 308 can render informationwindows 1104 that display relevant information about the automationsystem controlled by the cabinet 1102 (e.g., “Line 4” in the illustratedexample), as well as the panel-mounted devices within the cabinet.

In some embodiments, rendering component 308 can determine which subsetsof plant data 910 are associated with a given industrial asset beingviewed—and render this data in association with the assetaccordingly—based on either a determination that the data originatesfrom the industrial asset (e.g., industrial controller statusinformation stored on a data register of the industrial controller), orinformation stored on plant models 524 that defines associations betweendata items and industrial assets. In the latter scenario, associationsbetween data items (e.g., controller data tags, data values generated bymotor drives, measured values generated by telemetry devices, etc.) andindustrial assets can be defined by the user during configuration of theassets and/or the VR/AR presentations using a configuration application.The configuration application can be, for example, a bundled applicationthat includes both industrial device programming tools as well as VR/ARconfiguration tools, such that device configuration and visualization ofthe device's available data can be configured in parallel using the sameconfiguration tool. For example, during programming and configuration ofan industrial controller using the configuration application, the usermay define a data tag for storing a water pressure value (e.g., Tank 1Water Pressure) measured by a pressure meter associated with a tank(e.g., Tank 1). Plant models 524 may define information about Tank 1 forthe purposes of rendering VR or AR presentations for the tank, includingbut not limited to a location of the tank within the plant facility, agraphical representation of the tank, any animations associated with thetank, etc. After defining the Tank 1 Water Pressure data tag, theconfiguration application can also be used to specify that the datatag's value is to be associated with the Tank 1 entity. This associationdefinition will be stored in one or more of the plant models 524, andbased on this definition rendering component 308 will render the Tank 1Water Pressure data value on a user's wearable appliance 206 in responseto determining that the appliance's location and orientation places Tank1 (or a graphical representation of Tank 1 in the case of VRpresentations) within the user's line of sight.

In some embodiments, rendering component 308 can be configured todisplay a minimal amount of information about the cabinet 1102 (or othermachine or industrial device) in response to determining that thecabinet 1102 is within the user's current line of sight, and displayadditional information about the cabinet in response to a user gestureor verbal command (e.g., a natural language spoken command) indicating arequest for more detailed data. Basic information may include, forexample, a name of the machine or production line associated with thecabinet 1102, a current operating mode of the machine or line,production statistics such as a current product count or accumulateddowntime duration, current or historical fault information, or othersuch data. More detailed information that can be displayed for thecontrol cabinet 1102 can include, but is not limited to, a listing ofdevices contained in the cabinet, status or operational information forcontrol devices in the cabinet (e.g., run status for industrialcontrollers, operating statistics for a variable frequency drive,contactor open/closed statuses, etc.), an indication of whether accessto the corresponding physical cabinet requires arc-flash protection, orother such information, firmware revision information for devices withinthe cabinet 1102, network topology information, bill of materialschematics or electrical schematics for the cabinet 1102 (which may beretrieved by presentation system 302 from device documentationrepositories 510 or from other sources), vendor and model informationfor devices in the cabinet 1102, or other such information.

Similar information can also be invoked for industrial machines that aremonitored and controlled by the control cabinet devices. For example,while the user is viewing a virtual stamping press, rendering component308 can render a current operating mode of the press (e.g., running,idle, homed, etc.), a current operating speed of the press, a currentproduct count (e.g., a daily total or a total for the current shift),current or historical alarms or faults for the press, or other suchinformation. As described above, associations between the virtualstamping press (or other industrial asset or entity) and the subsets ofplant data 610 to be rendered in association with the press can bedefined by the user using a configuration application and stored inplant models 524.

In addition to presenting asset data to the user in response todetermining that the user's location and orientation places the assetwithin the user's line of sight, some embodiments of VR/AR presentationsystem can also process natural language spoken queries requestingspecified information about an industrial asset, regardless of whetherthe user is currently viewing the asset. For example, the user may speaka request for a current status of a particular asset (e.g., anindustrial robot, a production line, a motor, a stamping press, etc.),which is received by the user's wearable appliance 402 and relayed tothe VR/AR presentation system 302. The presentation system 302 cantranslate the spoken request into a query for the desired informationabout the specified asset, retrieve the relevant subset of plant data610, and render the requested information as a VR/AR presentation on theuser's wearable appliance 206.

In the control cabinet and stamping press examples described above (andin similar scenarios), rendering component 308 determines a suitablesubset of relevant information to display on the user's wearableappliance—as well as a location within the presentation at which thedata is to be displayed—based on the user's current location andorientation information (as obtained via location and orientation data606), as well as a determination of the user's current virtual locationwithin the simulated plant environment. For example, if the user isdetermined to be at a virtual location in front of a virtual controlcabinet, and is oriented such that the control cabinet is within theuser's line of sight, rendering component 308 will retrieve and renderinformation relevant to the cabinet and its associated devices at alocation within the presentation on or near the virtual cabinet. In someembodiments, rendering component 308 can determine which industrialassets are within the viewer's line of sight based on a correlation ofthe user's current location and orientation (as determined from thelocation and orientation data 606) with known locations of theindustrial assets making up the industrial facility, which may bedefined by plant model(s) 524.

Also, in some embodiments, presentation system 302 can generate layeredVR views, and allow the user to selectively enable or disable layers inorder to customize the presentation. For example, while a user isviewing a virtual control cabinet 1102 or machine via the wearableappliance 206, presentation system 302 can render a representation ofthe exterior of the cabinet or machine. In response to input orinstructions received via the wearable appliance 206 from the user(e.g., a gesture or natural language verbal command recognizable to thewearable appliance 206), presentation system 302 can remove layers fromthe presentation to expose interior views or layers of the cabinet 1102or machine, such that interior components of the cabinet or machine aredisplayed. This interior presentation can include graphicalrepresentations of physical components or conditions within the cabinetor machine (e.g., jams within a machine, moving parts within themachine, non-moving parts, etc.) as well as data presentations (e.g.,temperatures or temperature changes of interior machine or cabinetcomponents, imminent overcurrent indications, etc.).

The first-person VR/AR presentation described above can be useful if theuser is at a remote location relative to the physical production area(e.g., at another location within the plant or outside the plant), sincethe VR/AR presentation can be invoked from any location at which theuser's wearable appliance 206 can interface with the VR/AR presentationsystem 302. If the user is physically located within the actualproduction area, the system allows the user to switch to an AR/VRpresentation that renders data, graphics, or virtualized industrialassets over the user's direct view of real industrial assets. For suchVR/AR presentations, the rendering component 308 overlays virtualelements over the user's view of a real environment through the wearableappliance 206. In this way, as the user traverses the production area,the presentation system 302 can enhance the user's real-world view ofthe production area with data overlays comprising relevant subsets ofcollected plant data 610, as well as any relevant computed statisticsgenerated by reporting component 310.

Some information rendered in this VR/AR presentation can be similar toinformation presented during the remote first-person VR/AR presentationdescribed above. For example, in response to determining that the user'scurrent location and orientation indicates that the user is currentlyviewing a control cabinet via the wearable appliance, renderingcomponent 308 can generate and deliver an AR/VR presentation thatincludes relevant information about the devices within the cabinetand/or the automation system associated with the cabinet. In this way,the VR/AR presentation system 302 combines the user's line-of-sightinformation (e.g., the user's natural view, as determined based on theuser's location and orientation data 606) with application data relevantto the control panel, machine, or automation system that the user iscurrently viewing.

Also, in some embodiments, presentation system 302 can detect when auser is viewing a control cabinet (or a virtual control cabinet) throughthe wearable appliance 206, and present a virtual x-ray view of thecontents of the control cabinet, including a virtual view of thepanel-mounted devices within the control cabinet (e.g., industrialcontrollers, motor drives, contactors, etc.). The virtual view caninclude graphical representations of the devices, as well as relevantdata associated with one or more of the devices contained within thecabinet (obtained from plant data 610). In some embodiments, the usercan send a request to present system 302 (e.g., via a gesture or verbalcommand recognizable to the wearable appliance) for additionalinformation about the control cabinet, including electrical schematicsor line diagrams for the cabinet, ladder logic programming associatedwith an industrial controller mounted within the cabinet, diagnosticdata for any of the devices, etc. In response to the request, thepresentation system 302 will retrieve the requested data and render therequested data as an overlay on the user's view through the wearableappliance.

In some embodiments, information being viewed via a VR/AR presentationcan be selectively filtered by the user. For example, a bank of motordrives may generate and store a variety of operational and diagnosticdata (e.g., motor speed data, motor current data, alarm data, etc.).When a user is viewing the bank of motor drives (or a virtualrepresentation of the bank of motor drives) via the wearable appliance,the user may request—via a gesture or verbal command recognizable to thewearable appliance 206—a view that identifies which of the motor drivesrequires a fan replacement (e.g., based on a corresponding alarm that isactive on the drives). The user may also invoke a view that identifiesthe firmware versions currently installed on the respective drives, orthat identifies (e.g., using a graphical overlay) which of the drivesare currently configured with an outdated firmware version. Suchselective data filtering can be applied on substantially any parameteravailable within the devices or machines being viewed.

Rendering component 308 can also filter the data presented to the userbased on the user's identity or role, as defined by the user profiles522. In this regard, user profiles 522 may define the set of informationfor each machine or device that the user is allowed to view, andrendering component 308 can limit the data that is accessible by theuser to those defined sets of data. For example, for users having an“operator” role, rendering component 308 may only allow the user to viewdata relevant to operation of a machine or automation system (e.g.,operating modes, alarm information, running speeds, product counts,etc.). For users having an “engineering” role, rendering component 308may further allow the user to view firmware information for controldevices, industrial control programming (e.g., ladder logic or otherprogramming), network statistics, or other such engineering data.

In addition to rendering of VR/AR presentations on wearable appliance206, some embodiments of the VR/AR presentation system 302 can allowcontrol instructions to be originated from wearable appliance 206 anddelivered to a target control device (e.g., an industrial controller, amotor drive, a human-machine interface terminal, or other such controldevice). In scenarios in which the wearer of wearable appliance 206 isphysically located on the plant floor, such control instructions can bedelivered directly from the wearable appliance 206 to the control devicein some embodiments. FIG. 12 is a diagram illustrating data flowsbetween presentation system 302, wearable appliance 206, and anindustrial controller 1204 in a scenario in which wearable appliance isconfigured to direct control instructions 1206 to the industrialcontroller 1204. Similar to the data flows depicted in FIG. 7, VR/ARpresentation system 302 collects plant data 610 from a number ofindustrial devices throughout a plant facility, including industrialcontroller 1204. VR/AR presentation system 302 also obtains useridentity data 602 and location and orientation data 606 from wearableappliance 206 to facilitate generation of VR/AR presentation data 604,as described in previous examples. The layout of the VR/AR presentationsand the plant data integrated into the presentations are based in parton plant models 524 and user profiles 522.

In this example embodiment, device communication component 406 of thewearable appliance 206 supports a communication stack 1210 that allowsdirect communication between the wearable appliance 206 and industrialcontroller 1204 (and other industrial devices) via an industrial network(a wired or wireless network) on which industrial controller 1204resides. In an example implementation for use with CIP networks, thecommunication stack 1210 can support CIP protocol carried byEtherNet/IP. However, embodiments described herein are not limited tothese protocols. Through this direct communication between the wearableappliance 206 and industrial controller 1204 (or other automationsystems, machines, etc.) the user can send control information to theindustrial controller 1204 using gesture interactions with the ARpresentation or recognizable verbal commands.

In an example scenario, while the user's line of sight is directed tothe industrial controller, a control cabinet in which the controllerresides, or a machine being controlled by the industrial controller1204, the user can perform a gesture or speak a verbal commandrecognizable by the wearable appliance 206 indicating that the user hasselected the industrial controller 1204 (or its associated machine) as atarget for a control instruction. In response to the gesture or verbalcommand, wearable appliance 206 sends the identity of the target deviceto the VR/AR presentation system as selection data 1202. In someembodiments, selection of the industrial controller 1204 or itsassociated machine can cause rendering component 308 to render on theVR/AR presentation a predefined list of available commands that can beissued for the selected machine. Example commands can include, forexample, machine start/stop commands, switch setting adjustments,setpoint adjustments, alarm reset commands, or other such comments. Theuser can select from among the list of predefined commands using asuitable recognizable gesture or verbal command, and the wearableappliance 206 can issue the selected command to the industrialcontroller 1204. For example, if the command is a binaryinstruction—such as an alarm reset command, a start command, or a stopcommand—the wearable appliance 206 can direct a momentary or latching ONcommand to the appropriate register of the industrial controller's datatable via the plant network (e.g., a CIP network on which the controllerresides), causing a logical 1 bit to be written to the register. In thecase of commands requiring numeric input from the user—such as asetpoint adjustment—rendering component 308 can prompt the user to entera numeric value using recognizable gestures (e.g., by selectinggraphical up or down arrows to increase or decrease the numeric value)or by speaking the desired numeric value, and wearable appliance 206 cansend the entered numeric value as an analog value directed to theappropriate register of the controller's data table. Communication stack1210 of the device communication component 406 can send these controlinstructions 1206 via the wired or wireless network on which thecontroller 1204 resides, using communication stack 1210.

The available commands that can be selected and delivered to thecontroller for a given machine within the user's line of sight can bepredefined using a configuration application used to configure aspectsof the VR/AR presentations. In some embodiments, the configurationapplication can be a bundled software application that allows industrialdevices and their associated VR/AR presentations to be configured inparallel. In an example configuration scenario, the configurationapplication can be used to configure a read-write data tag of anindustrial controller representing a setpoint value, a command bit, oranother type of command value that can be written to the controller'sdata table as a control instruction. The controller's controlprogramming (also developed on the configuration application) may usethe value of the data tag as a user-adjustable setpoint (e.g., a targetfill level, speed, temperature, etc.), as an ON bit to trigger afunction or sub-routine, or as another type of command input. Using theconfiguration application, the user can identify the data tag as awritable tag capable of being written via the VR/AR presentation Theconfiguration application can also allow the user to associate the datatag with an identified machine, control cabinet, automation system, orother industrial asset. When such an association is defined between adata tag and an industrial asset using the configuration application,rendering component 508 will render data stored in the data tag on awearable appliance 206 when it is determined that the appliance'slocation and orientation places the asset within the wearer's line ofsight (provided the wearable appliance is associated with a user havingviewing privileges for the data item). If the data tag has beenconfigured as a tag to which the user can write commands via VR/ARpresentations, the rendering component 308 can also render a selectablecommand that can be issued to the data tag, as described above (e.g.,“adjust setpoint,” “adjust speed,” “switch to semi-auto mode,” etc.). Insome embodiments, the text of the available commands can be retrievedfrom metadata associated with the read-write data tag, such as a nameprovided to the tag using the configuration application (e.g., Semi-AutoMode On, Conveyor Speed, etc.).

Some embodiments that support direct communication between wearableappliance 206 and industrial devices (such as industrial controller1204) can also support bi-directional communication between the wearableappliance 206 and the industrial devices. In such embodiments, inaddition to receiving selected subsets of plant data 610 from VR/ARpresentation system 302 for rendering on the VR/AR presentation, livedata may also be retrieved directly from the industrial device bywearable appliance 206 via the control network (e.g., a CIP network) andrendered on the VR/AR presentation.

In some alternative embodiments, a VR/AR presentation for a givenindustrial asset can be retrieved and presented to a user's wearableappliance 206 in response to identification of the industrial asset bythe wearable appliance. FIG. 13 is a diagram illustrating an exampledata flow that facilitates communication between wearable appliance 206and one or more industrial devices 1304 for generation of virtualreality or augmented reality presentations. In this example, thewearable appliance 206 identifies an industrial device, machine,automation system, or control panel by scanning a quick response (QR)code 1306 associated with the industrial device or system to be viewed.The wearable appliance 206 then wirelessly sends the device or systemidentifier obtained from the QR code 1306 to VR/AR presentation system302. Additionally, the wearable appliance 206 can send authenticationinformation (e.g., user identity data 602) that can be used to identifyand/or authenticate the user of wearable appliance 206.

Based on the identity of the device or system that the user isrequesting to view, as well as the identity or role of the user, VR/ARpresentation system 302 can determine whether the user is authorized toreceive a virtual or augmented reality interface for the device orsystem, as well as a degree of control privilege for which the user isauthorized based on either the user's identity or the user's role. Forexample, depending on the user's identity or role, the user may begranted view-only privileges, or may alternatively be granted full orlimited control privileges whereby the user is permitted to delivercontrol instructions 1206 to the device or system.

Upon determining that the user is authorized to view and/or virtuallycontrol the device or machine, AR/VR presentation system 302 sends avirtual or augmented reality presentation to the wearable appliance 206.In an example scenario, the presentation can take the form of a virtualcontrol panel corresponding to the device identified by the QR code1306. The virtual control panel may be a predefined virtual panel havinga similar layout and functions as those of an actual control panelassociated with the device or system. Alternatively, the virtual controlpanel may be an entirely virtual panel with no analogous physicalcontrol panel. The virtual control panel can be stored on memory 322associated with VR/AR presentation system 302 and selected by the system302 based on the identity of the device or system and the controlprivileges of the user issuing the request.

Once the wearable apparatus 206 has received the augmented or virtualreality presentation, the user can remotely control the industrialdevice or automation system through interaction with the virtual controlpanel. For example, through the direct communication between thewearable appliance 206 and industrial devices 504 (or other automationsystems, machines, etc.) the virtual control panel can receive anddisplay status information for the devices, as well as send controlinformation to the devices (e.g., start/stop commands, switch settings,setpoint adjustments, alarm reset commands, etc.).

In the example depicted in FIG. 13, identification of the industrialdevice, automation system, or machine being viewed by the user isdetermined based on identification information obtained from the QRcode. However, in some embodiments the identity of the device or systemcan be determined by VR/AR presentation system 302 based on the user'slocation relative to the device or system, as well as the determinedorientation of the user's wearable appliance 206. For example, based onlocation and orientation data 606, presentation system 302 can infer theuser's current line of sight. By cross-referencing this location andorientation information with known location information for devices andsystems within the plant environment, presentation system 302 can inferwhich devices or systems are currently within the user's field of view.Presentation system 302 can then send suitable virtual or augmentedreality presentations (e.g., virtual control panels) to the wearableappliance 206 in accordance with the user's authorization and theidentities of the devices and systems currently being viewed. Thesepresentations can include both data generated by the devices or systemsbeing viewed, data generated by other devices or systems (e.g., analyticsystems, maintenance history databases, etc.) that has been defined ashaving an association with the device or system being viewed, or othersuch information.

In some embodiments, the system 302 can collect video (or audio-video)data from one or more cameras distributed throughout the plantenvironment, and integrate selected sets of this video data with a VR/ARpresentation. When 360-degree cameras are used, such embodiments canprovide users at remote locations with an interactive live video feed ofthe plant facility, simulating the user's physical presence on the plantfloor. FIG. 14A is a diagram illustrating an example configuration thatincorporates video information into VR/AR presentations generated by theVR/AR presentation system. In this example, a number of video capturedevices 1414 (e.g., digital video cameras or other types of videocapture devices) are installed at various locations throughout the plantfacility to capture video of respective areas of the facility. The videocapture devices 1414 are installed on a network having access to thehardware or cloud-based platform on which the VR/AR presentation system302 is implemented. In this example, each video capture devices 1414 is360-degree camera that generates video data 1412 for a full 360-degreeviewing angle surrounding the camera. Each video capture device 1414pushes the video data 1412 to the VR/AR presentation system 302 asindividual video streams. A video processing component 312 of the VR/ARpresentation system 302 can process and store each video stream on videostorage 1404 (which may be a portion of memory 322 or another datastorage device), such that the video data 1412 from each camera istagged with identification information indicating the source videocapture device 1414, time stamp information indicating a time at whicheach frame of the video data 1412 was recorded, and optionally the plantfacility and work area recorded by the video data 1412.

As shown in the example external (down-scaled) VR/AR presentationsdepicted in FIGS. 8-10, rendering component 308 renders camera icons 806(e.g., camera icons 806 a-806 d in FIGS. 8-10) corresponding to eachvideo capture device 1414 installed on the plant floor. In one or moreembodiments, the location of each video capture device 1414 within theproduction area or facility can be defined in one or more of the plantmodels 524, which can also specify the identity of the video capturedevice 1414 corresponding to each camera icon 806. In an example cameradeployment scenario, the user can interface with the VR/AR presentationsystem with a configuration application, which may render a browsableview of industrial devices currently networked to the VR/AR presentationsystem 302. After a video capture device 1414 has been mounted at adesired location and communicatively connected to the plant network, theuser can browse to video capture device 1414 via the configurationapplication, and add the video capture device 1414 to the current VR/ARproject. The location of the video capture device 1414 within the plantcan be entered manually by the user via the configuration tool, or maybe learned by device interface component 314 based on self-reportinglocation data received from the video capture device 1414 over thenetwork. This location information, together with the identity of thevideo capture device 1414, can then be saved to one or more of the plantmodels 524 so that this information can be referenced by renderingcomponent 308 in connection with rendering a corresponding camera icon806 at the proper location within a VR/AR presentation.

Based on this camera definition information defined in plant model(s)524, rendering component 308 places the camera icons 806 at locationswithin the VR presentation corresponding to the defined physicallocations of the video capture devices within the real plant facility.In some embodiments, video capture devices 1414 can be configured todetermine their own geographic location (or location relative topositioning devices within the plant), and report their identities andlocations to the VR/AR presentation system 302, which can then recordthe locations and identifies of each video capture device 1414 in plantmodel(s) 524 so that camera icons 806 can be rendered at the correctlocations. This technique can simplify integration of newly installedvideo capture devices 1314 with the VR/AR presentation system 302.

In an example application, one or more of the video capture devices maybe mounted on a stack light associated with a machine or production linein order to offer an elevated 360-degree view of the area surroundingthe machine or production line. FIG. 15 is another perspective of theVR/AR presentation depicted in FIGS. 8-10 that offers a closer view ofcamera icon 806 d. In this example, one of the video capture devices1414 has been mounted on a stack light in the physical plant, which isrepresented by virtual stack light 1502. As can be seen in FIG. 15,camera icon 806 d, which corresponds to this video capture device 1414,has been rendered over the virtual stack light 1502.

Although the examples depicted in FIGS. 8-10 and 15 depict the cameraicons 806 in the context of the external, down-scaled VR/ARpresentation, the camera icons 806 can also be made visible in thecontext of first-person VR/AR presentations.

In the example VR/AR presentations illustrated in FIGS. 8-10 and 15,camera icons 806 are spherical “bubble” shapes with an embedded camerashape to assist identification of the icons' function. In someembodiments, instead of or in addition to the embedded camera shape, thespherical camera icons 806 may contain a still or live video imageobtained for the corresponding video capture device 1414, therebyproviding the user with a preview of the video available at thatlocation. While viewing the VR/AR presentation, the user can select oneof the camera icons 806 using an appropriate selection gesture or verbalcommand in order to transition from the current external view of theproduction area to a live interactive video presentation rendered on thewearable appliance 206. With reference to FIG. 14A, in response to theselection gesture or verbal command, system interface component 404 ofwearable appliance 206 sends camera selection data 1402 to the VR/ARpresentation system 302. The camera selection data 1402 indicates theidentity of the video capture device 1414 selected by the user. As shownin FIG. 14B, in response to receipt of the selection data 1402,rendering component 308 transitions the user's current VR/ARpresentation view to a live video presentation 1408 supplied by thevideo capture device 1414 corresponding to the selected camera icon 806.The video presentation 1408 is generated based on camera-specific videodata 1410 retrieved from video storage 1404, which is the subset ofreceived video data 1412 corresponding to the selected video capturedevice 1414. In some implementations, video and audio from the selectedvideo capture device 1414 may be sent by the presentation system 302 tothe wearable appliance 206 via two separate communication channels ornetworks in order to satisfy bandwidth requirements on a given network.

Video presentation 1408 simulates the user's actual presence on theplant floor by rendering the camera's current perspective of theproduction area on the user's wearable appliance 206. If the video datais supplied by a 360-degree camera, rendering component 308 can trackthe orientation of the user's wearable appliance 206 and shift the videopresentation's direction of view in accord with the user's headmovements (left, right, up, or down). In this way, the segment of theavailable 360-degree video view available from the camera that is withinthe user's line of sight can be changed when the user moves his or herhead, simulating the real-world view the user would experience if theuser were at the camera location. Moreover, if the video capture devices1414 are configured to capture audio as well as video from thesurrounding areas, VR/AR presentation system 302 can receive the audiodata together with the video data 1412 from the video capture devices1414, and provide live audio together with the live video as part of thevideo presentation 1408. Other types of video capture devices 1414 canalso be used as sources for video presentations, including but notlimited to webcams, swivel-based IP cameras, etc.

In some embodiments, when the user selects a camera icon 806, renderingcomponent 308 can smoothly transition the user's current view (either anexternal view as shown in FIGS. 8-10 and 15, or a first-person VR/ARpresentation as shown in FIG. 11) to the live 360-degree videopresentation 1408 via a zoom action. For example, when the user selectscamera icon 806 d while the external view depicted in FIG. 15 is beingrendered on the wearable appliance 206, rendering component 308 willsmoothly zoom the user's view toward the selected camera icon 806 d.This zoom action simulates the user's physical transportation to thecamera. Once the zoom action causes the user's view to be encompassed bythe camera icon 806 d, rendering component 308 switches to the livevideo feed described above. Simulating the user's transportation to thecamera icon 806 d in this manner can help to orient the user within thevirtual production area, so that the transition to the video feed willnot disorient the user.

While viewing the video presentation, the user can use a suitablegesture or speak a verbal command recognizable by the wearable appliance206 to transition back to the VR/AR presentation when desired.

Some embodiments can also allow the user to transition the view to asecond video capture device 1414 while viewing a video presentationsupplied by a first video capture device 1414. In such embodiments,while the user is viewing a video presentation supplied by the firstvideo capture device 1414, if a second video capture device 1414 iswithin the viewing field or line of sight of the first video capturedevice 1414, rendering component 308 can overlay a camera icon 806 onthe video presentation at the location of this second video capturedevice 1414 (that is, on or near a video image of the second videocapture device 1414 within the video presentation). While viewing thefirst video presentation, the user can select this second camera icon806 using a recognizable selection gesture or verbal command Inresponse, the rendering component 308 will transition the user's videopresentation from the first video capture device 1414 to the selectedsecond video capture device. In this way, the user can easily “hop”between different camera video feeds that are within each other's lineof site. Also, in some embodiments, the system can allow the user totransition between any of the cameras installed in the production area,regardless of whether the cameras are within each other's line of sight.

Installation points for the 360-degree video capture devices 1414 can beselected to provide remote visibility into critical or dangerous areas.For example, it may be useful to install one or more video capturedevices 1414 near safety access points or near equipment that may posesafety risks, so that these areas can be visually monitored remotely.Video capture devices 1414 may also be installed at locations that aretypically unmanned but which may merit visual inspection. By integratingthese 360-degree video capture devices 1414 with VR/AR presentationsystem 302, these critical areas can be visually inspected from anylocation having access to the VR/AR presentation system 302.

Although the preceding examples have been described in terms ofdelivering live video feeds to the user's wearable appliance 206, someembodiments can also allow the user to access historical videorecordings stored in video storage 1404 by video processing component312. In such embodiments, the user can instruct wearable appliance 206to send a request to the VR/AR presentation system 302 identifying avideo capture device 1414 to be viewed, as well as a starting date andtime for the historical video to be viewed. In response to receiving therequest, rendering component 308 retrieves the stored video datacorresponding to the identified video capture device 1414 and theindicated date and time, and can begin streaming the retrieved videodata to the wearable appliance 206. Similar to the live video feeds, theuser can interact with the historical video feed by moving his or herhead to the left or right to change the perspective or line of site.

In some embodiments, the monitoring component 316 of VR/AR presentationsystem 302 can be configured to monitor the available video data 1412received from the video capture devices 1414, and generate alerts inresponse to detection of a possible issue that merits the attention ofplant personnel. In such embodiments, monitoring component 316 can betrained to recognize notifiable events within each stream of video data1412. This training can be customized to each stream of video data 1412,since events considered crucial are typically dependent upon the areabeing monitored. For example, if a video capture device 1414 is mountedat a high-security area of the facility within which human entry is tobe regulated, monitoring component 316 can be configured to recognizepresence of humans within the area based on analysis of the video data1412 received from that video capture device 1414. In some embodiments,monitoring component 316 can be configured to perform two-dimensionalimaging analysis on the video data 1412 and to recognize presence of ahuman based on a result of the imaging analysis. Other video analysistechniques can also be used without departing from the scope of one ormore embodiments.

In another example, a video capture device 1414 may be mounted at alocation near a critical machine so that the machine can be visiblyand/or audibly monitored. The monitoring component 316 can be trained torecognize one or more critical performance events based on analysis ofthe video data 1412 captured for the machine. For example, if themachine processes parts or products and is susceptible to jamming (e.g.,paper jams in a paper processing machine fed by a web tension system,part jams on a conveyor belt, etc.), monitoring component 316 can beconfigured to recognize occurrence of such jams based on analysis of thecorresponding video data. As in the case of human detection, jam eventsmay be detected based on imaging analysis of the video data 1412, orusing other video analysis techniques. Imaging analysis of the videodata 1412 can also be used to detect improper machine movements (e.g.,over-stroking or under-stroking, excessive vibration, etc.), presence ofexcessive smoke levels, or other events indicative of a possiblemaintenance issue. In another example application, multiple cameras maybe mounted at locations that capture a critical area from multipleangles, and monitoring component 316 can be configured to collectivelyanalyze images and/or audio from the various cameras. Such collectiveanalysis can be used to determine a location of a person, object, orvehicle within the monitored area using triangulation techniques, or forother purposes.

When video capture devices 1414 supporting audio capture are used,monitoring component 316 can also be trained to recognize audio queuesindicative of events requiring attention. For example, some machines mayemit noise at a recognizable characteristic frequency when experiencingexcessive vibration, or when the machine is otherwise running improperlyor in a sub-optimal manner. Monitoring component 316 can be configuredto recognize these audio queues based on analysis of audio-video datareceived from the video capture device 1414, and initiate delivery ofnotifications to appropriate personnel.

In addition to audio and video data, some embodiments of monitoringcomponent 316 can also be configured to analyze other types ofinformation received from video capture devices 1414 that support suchinformation. For example, one or more of the video capture devices 1414can support capture of infrared or ultraviolet data, and to provide thisinformation in addition to or as an alternative to video data 1412.Monitoring component 316 can analyze this information to determinewhether a temperature of a machine or a mechanical component isexcessive. In another example, one or more of the video capture devices1414 may be a time-of-flight (TOF) optical scanner or sensor, whichgenerates distance information (e.g., point cloud or depth mapinformation) for objects and surfaces within the scanner's field ofview. In such embodiments, monitoring component 316 can be configured tocorrelate object recognition results with the distance information, andgenerate a notification directed to a wearable appliance or anindustrial controller in response to determining that a result of thiscorrelation satisfies a defined criterion.

In an example TOF application, monitoring component 316 may beconfigured to detect, based on analysis of TOF distance data obtainedfrom a TOF optical sensor, that an object detected in the video data1412 is located at a distance from the video capture device 1414 that isless than a defined safe distance. In some embodiments, monitoringcomponent 316 may also determine a type of the object based on imaginganalysis of image data obtained from the TOF optical sensor or fromanother image capture device. If the type of the object corresponds toan object type that is not permitted to be within the defined safedistance (e.g., a human, a vehicle, etc.), and the TOF sensor dataindicates that the object is within the defined safe distance, themonitoring component 316 can execute a defined action. The action may befor example, setting a bit or register in an industrial controller thatalters an operation of a machine (e.g., by placing the machine in a safestate, such as a stopped or slowed state). The action may also includeoverlaying a notification graphic on the video data generated by thevideo capture device to indicate a location within the image at whichthe object was detected, as well as an alphanumeric message notifying ofthe possible security or safety issue.

In response to detection of a notifiable event (such as those describedin the examples above), monitoring component 316 can instruct renderingcomponent 308 to deliver notification information to the wearableappliances 206 associated with one or more users. The target users to benotified can be determined based on one or more of the type of issuedetected, the area in which the issue is detected, an identification ofwhich users are within a defined distance from the source of the issue(e.g., based on analysis of the location and orientation data 606received from the wearable appliances 206), or other suchconsiderations. rendering component 308 may also consider informationcontained in the user profiles 522 when determining a suitable subset ofusers to be notified. For example, user profiles 522 may define theroles of each user, as well as which production areas of a plantfacility each user is assigned to. If the issue is determined to requireattention by maintenance personnel, rendering component 308 canidentify, based on analysis of the user profiles 522, which users areboth identified as maintenance personnel and are assigned to theproduction area in which the issue was detected.

Notifications can take the form of a superimposed message rendered onthe user's wearable appliance 206 identifying the nature of the issue.In some embodiments, if the user is located on the plant floor at thetime of the notification, rendering component 308 can render a VR/ARpresentation that superimposes directional arrows over the user'snatural view of his or her environment directing the user to the sourceof the issue. The directional arrows may first guide the user to themachine or area at which the issue was detected. The direction of thearrows, as well as the location of the arrow graphics on the displayscreen of the wearable appliance 206, are a function of the user'scurrent location and orientation, as determined by the location andorientation data 606. Once at the location, further directional arrowscan be generated that indicate the particular industrial device,machine, or machine component experiencing the issue. Again, thedirection and display locations for these arrows are based on thecurrent location and orientation data 606. As the user changes locationand orientation, rendering component 308 will update the directionsand/or display locations of the arrows and other graphical indicators inaccordance with the updated location and orientation data 606 to ensurethat the graphical indications continuously direct the user's attentionin the correct direction or toward the correct devices or components.

As will be discussed in more detail below, detection of an event canalso initiate delivery of interactive workflow data to the notificationrecipient's wearable appliance 206. This workflow data can guide theuser, or multiple users working in conjunction, through the process ofcorrecting the identified issue via interactive workflow presentations.

In some embodiments, notifiable events can also be detected based onanalysis of plant data 610. In various examples, critical operatingparameters, such as temperatures, pressures, speeds, voltage levels, orother such parameters collected as part of plant data 610 can bemonitored by monitoring component 316 to identify when these parametersfall outside defined acceptable value ranges. Monitoring component 316can also be configured to initiate notifications in response to machineor device alarm conditions detected in the plant data. Similar to eventsidentified via video analysis, rendering component 308 can generate anddeliver notifications to selected personnel in response to detection ofan issue based on analysis of the plant data 610.

As noted above, some embodiments of VR/AR presentation system 302 can beconfigured to assist users in connection with addressing detectedoperational or maintenance issue using interactive workflowpresentations customized for the detected issue. FIG. 16 is a diagramillustrating example data flows between VR/AR presentation system 302,industrial equipment, and wearable appliance 206 for delivery ofinteractive workflow presentations. The example illustrated in FIG. 16depicts a scenario in which the system 302 selects and delivers aworkflow presentation to a user's wearable appliance 206 dynamically inresponse to a detected condition.

As the device interface component 314 collects and indexes operationaland status data 1606 from industrial devices and systems on the plantfloor, as described above, monitoring component 316 monitors selecteddata items of the plant data 610, and initiates delivery of a suitableworkflow presentation 1602 in response to determining that one or moredata items indicate a problem with an automation system or device thatmerits attention from one or more registered users. For example, basedon current status and/or operational information for one or more of theindustrial devices (e.g., operating parameters, KPIs, etc.), monitoringcomponent 316 can detect when an industrial device or system hasgenerated an alarm or fault, experienced a downtime condition, performedan out-of-sequence operation, or other such condition. Monitoringcomponent 316 can also detect when that a performance metric of anindustrial process or machine (e.g. a KPI or other type of metric) hasdeviated outside an acceptable tolerance range, signifying a drop inproduction efficiency that may be correction through user intervention.As discussed above, monitoring component 316 can also detect an issuerequiring notification based on analysis of video data 1412 receivedfrom one or more video capture devices 1414.

For industrial controllers that monitor and control operations of anindustrial machine or process, the notification event detected by themonitoring component 316 may relate to the controller's internaloperation (e.g., a controller fault) or to the machine or process beingcontrolled. In the latter scenario, the alarm or fault conditionsassociated with the controlled machine or process may be predefined aspart of the control program being executed on the industrial controller.For example, process parameter setpoint values, abnormal machinestatuses, process alarm conditions, and other such notifiable conditionsmay be defined by a programmer within the industrial control program,and such conditions will be detected by the monitoring component 316 andused as the basis for a notification trigger.

Other types of industrial assets, such as telemetry devices, motordrives, etc., may have a different set of associated notifiableconditions that will be monitored by the monitoring component 316. Forexample, in the case of a motor drive (e.g., a variable frequency driveor other type of drive), the monitoring component 316 may monitor forinternal drive abnormal conditions, including but not limited toovercurrent or undercurrent faults, over-speed or under-speed faults,over-voltage or under-voltage faults, etc.

To facilitate generation of workflow presentations for assistance withdetected issues, VR/AR presentation system 302 can store (e.g., onmemory 322) workflow data 1608 defining actions to be taken to correctvarious issues, as well as VR/AR presentation instructions for renderingguidance in connection with performing these actions. In one or moreembodiments, sets of workflow data 1608 can be stored in associationwith the event or machine to which the workflow relates. For example, aset of workflow data 1608 may define a workflow determined to beeffective for recovering from a particular alarm condition of a conveyorsystem in association with the relevant alarm, so that the workflow canbe delivered to a user's wearable appliance 206 as a VR/AR presentationin response to detection of the relevant alarm. Similarly, workflowsassociated with preferred operation of a given automation system can betagged with an identifier of the relevant automation system, so that theworkflow will be delivered to wearable appliances 206 in associated withusers determined to be currently operating the relevant system.

In response to detection of an issue for which a defined workflow isavailable, the monitoring component 316 can send an instruction 1610 tothe rendering component 308 identifying the detected issue, whichinitiates delivery of a suitable workflow presentation 1602 to wearableappliances 206 associated with one or more users determined to becapable or authorized to address the detected issue. The instructionsent by the monitoring component 316 may include a subset of plant data610 that identifies the detected event and/or relevant industrialsystem. Based on this information, the rendering component 308 selects aset of workflow data 1608 associated with the identified event and/orautomation system, and delivers workflow presentation data 1602 to oneor more selected wearable appliances 206.

As noted above, rendering component 308 can identify one or moresuitable recipients for the workflow presentation based on the type ofthe event and/or the affected machine or device. In this regard, VR/ARpresentation system 302 can identify suitable recipients based on storednotification rules. These notification rules can comprise, for example,rules regarding which types of users or user roles should receivenotifications and workflows for different categories of events,restrictions on the types of data that can be presented to each userbased on the user's role, location-based restrictions on datapresentation, how workflow data should be presented for each type ofuser, etc. In some embodiments, rendering component 308 may narrow thelist of suitable recipients further based on current user contextinformation, including but not limited to each potential recipient'scurrent availability or location relative to the source of the detectedissue (as determined based on the location and orientation data 606received from the users' wearable appliances 206), skills or training ona particular device or piece of equipment to which the notificationrelates (as determined based on the user profiles 522), etc. In anexample scenario, rendering component 308 may determine each potentialrecipient's current location by tracking each user's location andorientation data 606, and deliver notifications and workflowpresentations only to those users within a defined radius of theaffected machine or device.

In some embodiments, identification of the suitable recipients can belearned by the system 302 as part of a workflow learning routine. Forexample, VR/AR presentation system 302 observes over multiple instancesof a particular machine downtime condition that certain specificemployees typically congregate to address the issue (as determined basedon monitoring of the wearable appliances 206 for the respectiveemployees), rendering component 308 may link these identified employeeswith the learned workflow associated with this downtime event, andmodify the notification rules to reflect this association. In someembodiments, if various personnel are observed to address differentoccurrences of the downtime condition, rendering component 308 mayfurther determine which of the personnel typically recover the machinein the least amount of time relative to other technicians. In accordancewith this determination, rendering component 308 may prioritize deliveryof subsequent downtime notifications and corresponding workflows to thisuser.

When all eligible recipients have been identified, rendering component308 send workflow presentation data 1602 to each recipient's wearableappliance 206. In an example scenario, rendering component 308 canrender the workflow presentation as an augmented reality presentationthat renders a sequence of instructions as an overlay on the user'sfield of view. These presentations can include graphical indicatoroverlays that point to or otherwise visually identify devices,workstations, or machine components that the user's attention should befocused on during a current step of the workflow, alphanumericinstructions that inform the user of the next step to be performed,feedback graphics that indicate when the step has been correctlyperformed or when the user has deviated from the proper workflowsequence, and other such information. Workflow presentations may includeboth alphanumeric instructions, as well as graphical guides thatillustrate certain steps of the workflow. These graphical guides mayinclude, for example, diagrams illustrating the action to be performed,photographic or video data that demonstrates how a given step is to beperformed, device documentation, or other such guides.

In general, workflow presentations inform the user of the propersequence of operations to be performed in order to best address thedetected condition. In some embodiments, while the detected condition isbeing addressed by a recipient of the workflow presentation, monitoringcomponent 316 can continuously compare actual operator actions with theoptimal workflow represented by the workflow data, and provide feedbackto the user if the user's actions deviate from the optimal workflow.FIG. 17 is a diagram illustrating real-time workflow feedback 1712 beingdelivered to a wearable appliance 206 as the user is correcting adowntime issue. During diagnosis and correction of the downtime issue,monitoring component 316 continues monitoring relevant subsets of plantdata 610 received from the automation system as the user carries outsteps to correct the issue and recover the system.

Concurrently, device interface component 314 collects user data 1704that can be used to confirm that the user is performing the stepsrecommended by the workflow delivered to the user's wearable appliance206. User data 1704 can include, for example, the user's identity andlocation relative to the automation system or components thereof. Theuser's location can be used to confirm that the user is at theappropriate location to perform the workflow step currently awaitingcompletion (e.g., in front of the appropriate control panel, HMI,machine station, or mechanical/electrical component). User data 1704 canalso include data indicating the user's interactions with devicesassociated with the automation system. Some of these interactions may beinferred based a correlation of the user's location relative to theautomation system and status information collected from one or moredevices of the automation system. For example, based on the user'sdetermined proximity to a control panel and a transitioning of a modeswitch on the control panel from a first position to a second position,monitoring component 316 can confirm that the user has placed theautomation system in the correct mode in accordance with a currentpending step of the workflow.

Other user data 1704 may include device orientation data identifying acurrent orientation of the user's wearable appliance 206, which, whencombined with the user's current location data, can indicate whether theuser is currently viewing a correct area of the automation system forcompletion of a pending workflow step. In some embodiments, the deviceinterface component 314 may also receive environment data collected bythe user's wearable appliance 206 in the form of multimedia (e.g., audioand/or video data); infrared data; heat signature data; vibration data(which may be obtained by the wearable appliance 206 via the user's bodywhen the user touches a vibrating component of an industrial machine orsystem); ambient noise levels; flux data; data indicative of thepresence of particular gases, particulates, smoke, or toxins within theuser's immediate environment; or other such environmental data. Suchenvironmental data may provide further information that can be leveragedby monitoring component 316 to determine if the workflow is beingfollowed, or if an action performed by the user in connection withaddressing the issue has produced an unexpected result of which the usershould be notified (e.g., overheating of a part, release of a toxin,elevated levels of smoke or particulates, etc.).

Based on a comparison of the user's interactions with the automationsystem with the steps of the preferred workflow, rendering component 308can generate and deliver workflow feedback data 1712 to the user'swearable appliance 206 in response to determining that the user hasdeviated from the workflow. Such feedback may comprise, for example,corrective instructions intended to inform the user of the deviation andto guide the user to the correct sequence of operations dictated by theworkflow. In some embodiments, monitoring component 316 can alsocalculate and record performance metrics for the user that rate theuser's performance of the workflow, based on the user's measured degreeof compliance with or deviation from the workflow. These performancemetrics can be based on such factors as a number of detected deviationsfrom the workflow, an average speed at which the user completesworkflows, a number of workflows carried out by the user, or other suchfactors. These performance metrics can be recorded in the user profile522 corresponding to the user. These metrics can be used by managementstaff to determine which operators require additional training inconnection with addressing specific performance or maintenance issues.Each user's performance metrics can also be included as part of theoperator information rendered in response to selection of a user'soperator information icon 804, as described above in connection withFIGS. 8-10.

In the case of collaborative action in which multiple users areaddressing a detected issue, rendering component 308 can deliverworkflow presentation data 1602 to each recipient's wearable appliance206 to coordinate activity between the recipients. The workflowpresentation displayed on each user's wearable appliance 206 will be afunction of the user's current location and direction of view. Forexample, if a first user is at a location corresponding to a step in aworkflow (e.g., placement of a production line in semi-auto mode, whichrequires the user to be in front of the control panel for the productionline), and a second user is at a location corresponding to another stepin the workflow (e.g., removal of a workpiece from a conveyor),rendering component 308 will render, on each user's wearable appliance206, the step of the workflow capable of being carried out by that userbased on the user's location and line of sight. When a step is completedby one user, rendering component 308 will update the workflowpresentations delivered to the other users to reflect completion of thestep.

To further facilitate coordination of activities between multiple usersaddressing a detected issue, VR/AR presentation system 302 can allowusers to share their personal views with one another. For example, afirst user on one side of a machine may wish to share his or her view ofthe machine with a second user on the other side of the machine or atanother location. In response to a suitable gesture or verbal commandreceived via the first user's wearable appliance 206, a live feed of theuser's current view through the wearable appliance 206 can be sent—viaVR/AR presentation system 302—to the second user's wearable appliance206. The second user can select whether the first user's view ispresented as a picture-in-picture video image, or as a “full screen”video whereby the first user's view is fully reproduced in the seconduser's viewing field as a video stream. Since the wearable appliances206 support audio communication, the users can exchange verbalcommunication via the wearable appliances 206 while sharing views inorder to facilitate coordination of activities between the users.

Although the foregoing example describes delivery of workflows, as wellas workflow feedback, in connection with a machine downtime event, alarmevent, or other maintenance actions, similar techniques can be used todeliver workflows to machine operators for carrying out specific machinecontrol operations during normal runtime of the machine, and to rate theoperator's compliance with the workflow. For example, if VR/ARpresentation system 302 has learned a preferred operator workflow forachieving optimal product throughput for a given production line, thisworkflow can be delivered to a wearable appliance 206 associated with anon-shift operator in response to determining that a performance metricof the production line has deviated outside an acceptable tolerance,thereby providing an operating methodology to the user for bringing theperformance metric back into acceptable tolerances.

Some embodiments of VR/AR presentation system may also allow individualusers to customize workflows and to save the customized workflows backto the system 302 in association with their user identity. For example,the VR/AR presentation rendered on the user's wearable appliance 206 mayinclude controls that allow the user to hide or remove one or more stepsof the workflow that the user finds unnecessary or unhelpful, therebyresulting in a personalized workflow for the given condition. The usercan then save this personalized workflow back to presentation system302, such that, when another instance of the maintenance or operationalissue occurs, the presentation system 302 will provide the user with themodified, personalized version of the workflow rather than the defaultversion. This can afford users a degree of control of the amount ofworkflow information that is provided via their personalized versions ofthe workflows. A user may choose to hide certain steps of a workflow asa function of the user's degree of experience or comfort level inaddressing the identified issue.

Also, in some embodiments, VR/AR presentation system 302 can allow usersto associate virtual notes or information tags with selected industrialassets for viewing by other users. These virtual notes can be used toconvey personalized information relating to the asset to other users whomay later view the asset via their wearable appliances 206. In anexample scenario, a maintenance person in the process of performingmaintenance on a conveyor system may wish to inform operators who willbe working a subsequent work shift that the conveyor should only be runat low speeds until future maintenance can be performed. The maintenanceperson may also wish to provide information to other maintenancepersonnel coming in during the subsequent shift regarding the workperformed during the current shift. VR/AR presentation system 302 canallow the user to associate virtual notes with the control cabinet forthe conveyor system, and to select which user roles are permitted toview each note. Accordingly, the user can compose a note for thenext-shift operators by speaking the text of the note into the wearableappliance 206, and instructing rendering component 308 to associate thenote with the selected control cabinet (e.g., by performing a gesture orverbal command corresponding to the instruction). The user can furtherinstruct rendering component 308 to only allow users of all roles toview the note. Accordingly, when rendering component 308 determines thatanother user's location and orientation data 606 indicates that the useris viewing the control cabinet, the rendering component 308 can renderthe note on the user's wearable appliance 206 as an overlay on or nearthe user's view of the control cabinet. Alternatively, renderingcomponent 308 can render a virtual note icon on or near the controlcabinet indicating that a virtual note has been attached to the controlcabinet. Selection of the virtual note icon using a suitable gesture orverbal command recognizable by the viewer's wearable appliance 206 cancause the text of the note to be displayed.

Similarly, the maintenance person can compose another virtual notedirected only to other maintenance personnel identifying maintenanceactions that were performed on the current shift, and remainingmaintenance actions that should be performed on the next shift.Accordingly, the maintenance person can compose the virtual note,associate the note with the control cabinet, and instruct that onlyother users having a maintenance role are permitted to view the note.Subsequently, in response to determining that a user having amaintenance role is viewing the control cabinet, rendering component 308will render the note (or a selectable virtual note icon) on the wearableappliances 206 associated with user.

Virtual notes can be used to share substantially any type of informationamong users, and can be selectively directed to all users, users of aspecified role, or specified individual users. Example messages that canbe conveyed via virtual notes can include, but are not limited to, alist of action items to be performed by operators or maintenancepersonnel working subsequent shifts, instructions to visually monitor amachine component that has been behaving erratically, or other suchmessages.

In some embodiments, the VR/AR representation of an industrial factorygenerated by VR/AR presentation system 302 can be used as the basis fora digital twin of the factory. In such embodiments, the plant models 524can not only model the physical appearance of industrial assets, but canalso model certain behaviors of those assets (e.g., responses to controlinputs in terms of movement, speed, temperatures, flows, fill levels,etc.). This can allow the VR/AR representation to act as a simulationenvironment for testing control programs or device configurations. FIG.18 is a generalized block diagram illustrating interactions between acontrol program 1802 being tested and a VR/AR model 1804 of anindustrial facility generated by rendering component 308. The VR/ARmodel 1804 is generated in a similar manner to the VR/AR presentationsdescribed above, based on plant models 524 and collected plant data. Inthis example, a simulation component 318 of the VR/AR presentationsystem 302 acts as an industrial controller emulator to execute controlprogram 1802 against VR/AR model 1804.

VR/AR model 1804 can simulate various aspects of a physical industrialsystem to be monitored and regulated by the control program 1802.Simulation component 318 can virtually interface control program 1802with the VR/AR model 1804 to exchange I/O data in order to simulatereal-world control. Control program 1802 can comprise any conceivabletype of code used to process input signals read into a controller and tocontrol output signals from the controller, including but not limited toladder logic, sequential function charts, function block diagrams, orstructured text. Control program 1802 is designed to regulate anautomation system being modeled by VR/AR model 1804. VR/AR model 1804mathematically models the system to be regulated by generating digitaland analog I/O values representing, for example, sensor outputs,metering outputs, or other plant data analogous to the data expected tobe generated by the physical system being modeled. These inputs andoutputs can be defined for each industrial asset by plant models 524.This simulated output data 1808 is provided to the simulation component318 executing control program 1802, which receives this data as one ormore virtual physical inputs. Control program 1802 processes theseinputs according to user-defined algorithms, and generates digitaland/or analog controller output data 1806 based on the processing. Thisoutput data 1806 represents the physical outputs that would be generatedby a controller executing control program 1802 and transmitted to thehardwired field devices comprising the automation system (e.g., PID loopcontrol outputs, solenoid energizing outputs, motor control outputs,etc.). The controller output data 1806 is provided to the appropriateinput points of the VR/AR model, which updates the simulated output data1808 accordingly. This simulation technique can be used to test anddebug control programs without putting field equipment and machinery atrisk, to simulate modifications to plant or machine operations andestimation how such modifications affect certain performance orfinancial metrics, or to perform other analytics.

Simulation component 318 can be configured to simulate execution ofmultiple control devices, including but not limited to industrialcontrollers, motor drives, and other such control devices. As moresimulated control devices are integrated with the VR/AR model 1804, adigital twin of the physical automation system can be realized. Thisdigital twin can be used to test new control programs on virtualizedequipment analogous to their real-world counterparts, perform predictiveanalytics to estimate asset maintenance or replacement schedules, orother such functions.

In some embodiments, VR/AR presentation system 302 can collect data froma wearable appliance 206 as the wearer traverses the plant floor, anduse this collected data to generate documentation of the plantenvironment. This can include, for example, creating piping andinstrumentation (P&ID) diagrams, device inventory data, machineinventory data, or other such plant mappings. In some embodiments, VR/ARpresentation system 302 can generate this documentation in athree-dimensional format. In some scenarios, the system 302 can collectthe data required to generate these plant mappings from multiplewearable appliances 206 associated with multiple users distributedthroughout the plant environment, and update the plant documentation asmore data is received.

In some embodiments, VR/AR presentation system 302 can create a virtualcontrol panel for an automation system or workstation, and present thisvirtual control panel to the user's wearable appliance 206. Someembodiments can dynamically generate the virtual control panel based onvisual information of the actual control panel collected by the wearableappliance 206 as the user is viewing the control panel. For example, asthe user is viewing a control panel, the wearable appliance 206 canobtain a visual snapshot of the panel, identify the locations of thebuttons, switches, or other control components on the panel, andgenerate a virtual representation of the control panel thatsubstantially duplicates the layout of the actual control panel. Theuser can then virtually interact with this virtual control panel inorder to perform remote control of the automation system with which thepanel is associated.

In some embodiments, the presentations generated by VR/AR presentationsystem 302 can also replace or enhance the human interface modules(HIMs) of motor drives. In such embodiments, when a user is viewing amotor drive through the wearable appliance 206, presentation system 302can render a presentation on the wearable appliance 206 that includesoperational and/or diagnostic statistics for the drive. Such statisticscan include, but are not limited to, motor speed, frequency, current,alarm or fault information (e.g., overcurrent faults), or other suchinformation.

Also, in some embodiments, the wearable appliance 206, working inconjunction with system 302, can be used to perform asset trackingfunctions. In this way, the wearable appliance 206 can be used as analternative to traditional asset tracking techniques (e.g., assettracking techniques based on radio frequency identification). In arelated aspect, some embodiments of presentation system 302 can assist auser in locating an asset (e.g., a drive, a sensor, a controller, etc.).In such embodiments, the user may speak or otherwise provide an assetidentifier of an asset to be located, and the wearable appliance 206will relay the asset identifier to presentation system 302. In response,presentation system 302 can deliver a presentation to the wearableappliance 206 that guides the user to the requested asset. For example,if the asset is within the user's field of view, the presentation cangenerate a highlight graphic on or near the asset. If the asset iselsewhere in the plant, the presentation can guide the user to the assetusing appropriate graphical indicators (e.g., arrow graphics, text-baseddirections, etc.). Presentation system 302 can accept the assetidentifier as a spoken or otherwise entered asset identifier number, oras a spoken or entered asset name. The user may also enter a request tolocate all assets corresponding to a particular asset type; e.g., byspeaking “show me all drives” or “show me all sensors.” In response,presentation system 302 will provide a presentation that highlights, ordirects the user to, all assets corresponding to the requested assettype.

FIGS. 19-22 illustrate various methodologies in accordance with one ormore embodiments of the subject application. While, for purposes ofsimplicity of explanation, the one or more methodologies shown hereinare shown and described as a series of acts, it is to be understood andappreciated that the subject innovation is not limited by the order ofacts, as some acts may, in accordance therewith, occur in a differentorder and/or concurrently with other acts from that shown and describedherein. For example, those skilled in the art will understand andappreciate that a methodology could alternatively be represented as aseries of interrelated states or events, such as in a state diagram.Moreover, not all illustrated acts may be required to implement amethodology in accordance with the innovation. Furthermore, interactiondiagram(s) may represent methodologies, or methods, in accordance withthe subject disclosure when disparate entities enact disparate portionsof the methodologies. Further yet, two or more of the disclosed examplemethods can be implemented in combination with each other, to accomplishone or more features or advantages described herein.

FIG. 19A is a first part of an example methodology 1900A for creatingand rendering VR/AR presentations of industrial facilities with theoption to transition to live video feeds. Initially, at 1902, data isreceived from a set of industrial assets installed an industrialfacility. The data can comprise, for example, operational or status datareceived from industrial controllers, motor drives, telemetry devices,sensors, or other such industrial devices. The data can also compriseconfiguration information for one or more of the industrial devices.

At 1904, a VR/AR presentation of the industrial facility is generated ona wearable appliance based on the data received at step 1902 and one ormore plant models that define graphical representations of theindustrial assets. The VR/AR presentation can offer either an externalview of the facility that renders an industrial area within the facilityas a virtual scale model of the facility with which the user caninteract, or a first-person view in which the industrial area isup-scaled to simulate the user's presence within the actual facility. At1906, camera identification data is received that identifies locationsof one or more 360-degree video cameras installed within the industrialfacility. At 1908, one or more camera icons corresponding to the one ormore 360-degree video cameras are rendered on the VR/AR presentationgenerated at step 1904, where the camera icons are positioned in theVR/AR presentation at locations corresponding to the locationsidentified by the camera identification data received at step 1906. Insome embodiments, the camera icons can comprise virtual spheres orbubbles. In some such embodiments, a preview of the video available fromeach camera can be rendered within each of the camera icons. At 1910,video data from the one or more 360-degree video cameras is received andstored.

The methodology continues with the second part 1900B illustrated in FIG.19B. At 1912, a determination is made as to whether selection input isreceived from the wearable appliance identifying a selected camera iconof the one or more camera icons. If no such selection input is received(NO at step 1912), the methodology returns to step 1902 and steps1902-1912 repeat. Alternatively, if such selection input is received(YES at step 1912), the methodology proceeds to step 1914, where therendering of the VR/AR presentation on the wearable appliance is zoomedfrom a current virtual view of the industrial facility toward the cameraicon identified by the selection input received at step 1912.

At 1916, a determination is made as to whether the zoom toward thecamera icon is complete. The zoom can be considered complete when theperspective offered by the VR/AR presentation is sufficiently close tothe camera icon that the icon encompasses a defined percentage of theuser's view. Other criteria for determining completion of the zoom arealso within the scope of one or more embodiments. If the zoom is not yetcomplete (NO at step 1916), step 1914 continues to execute the zoom.Alternatively, if the zoom toward the icon is complete (YES at step1916), the methodology proceeds to step 1918, were the rendering on thewearable appliance is changed from the VR/AR presentation to a videostream presentation comprising a subset of the video data received atstep 1910 from the 360-degree video camera corresponding to the cameraicon selected at step 1912.

The methodology continues with the third part 1900C illustrated in FIG.19B. At step 1920, a determination is made as to whether an instructionis received from the wearable appliance to transition back to the VR/ARpresentation. If no such instruction is received (NO at step 1920), themethodology proceeds to step 1922, where the video presentation isupdated on the wearable appliance in accordance with new video datareceived from the 360-degree video camera. At 1924, a determination ismade as to whether an indication is received from the wearable applianceindicating that the wearable appliance has changed orientation. If nosuch indication is received (NO at step 1924), the methodology returnsto step 1920. Alternatively, if such an indication is received (YES atstep 1924), the methodology proceeds to step 1926, where the videopresentation is updated on the wearable appliance to change thedirection of sight of the video presentation in accordance with the neworientation of the wearable appliance. The methodology then returns tostep 1920.

If an instruction to transition back to the VR/AR presentation isreceived from the wearable appliance (YES at step 1920), the methodologyproceeds to step 1928, where the rendering on the wearable appliancechanges from the video stream presentation to the VR/AR presentation ofthe industrial facility. The methodology then returns to step 1902 andrepeats.

FIG. 20A is a first part of an example methodology 2000A for controllinga VR/AR presentation of an industrial facility. Initially, at 2002, datais received from a set of industrial assets installed an industrialfacility. The data can comprise, for example, operational or status datareceived from industrial controllers, motor drives, telemetry devices,sensors, or other such industrial devices. The data can also compriseconfiguration information for one or more of the industrial devices.

At 2004, an external view of a VR/AR presentation of the industrialfacility is generated on a wearable appliance based on the data receivedat step 2002 and one or more plant models that define graphicalrepresentations of the industrial assets. The external view of thefacility renders the facility or a production area within the facilityas a virtual scale model with which the user can interact. For example,the perspective or angle of view of the scaled facility will change incoordination with the user's position and orientation to simulatewalking around a physical scale model of the facility.

At 2006, at least a subset of the data received at step 2002 is renderedon the VR/AR presentation. For example, production statistics or KPIsfor the facility or production area can be rendered near relevantvirtualized industrial assets within the virtual reality presentation.In another example, alarm information generated by an industrialcontroller or HMI can be rendered on the VR/AR representation on or nearthe industrial assets to which the alarms relate.

At 2008, a determination is made as to whether an instruction isreceived from the wearable appliance to render the VR/AR presentation ina first-person mode. If no such instruction is received (NO at step2008), the methodology returns to step 2002 and steps 2002-2008 repeat.Alternatively, if an instruction to render the VR/AR presentation in thefirst-person mode is received (YES at step 2008), the methodologyproceeds to step 2010, where the external view of the VR/AR presentationis transitioned to a first-person view in which the virtual scale modelis up-scaled to a size and perspective that simulates the user'spresence at the industrial facility.

The methodology continues with the second part 2000B illustrated in FIG.20B. At 2012, the first-person VR/AR presentation rendered at step 2010is updated with new data received from the set of industrial devices andorientation information received from the wearable appliance. Thisupdated can comprise, for example, updating values of operational andstatus data rendered on or near virtualized industrial assets within theuser's line of sight, as well as changing the perspective or line ofsight of the presentation in accordance with changes in the user'sorientation (e.g., moving forward or backward, or turning his or herhead left or right).

At 2014, a determination is made as to whether an instruction isreceived from the wearable appliance to render the VR/AR presentation asthe external view. If no such instruction is received (NO at step 2014),the methodology returns to step 2012 and the first-person presentationcontinues to update. Alternatively, if an instruction to render theexternal view is received from the wearable appliance (YES at step2014), the methodology proceeds to step 2016, where the first-personview is transitioned back to the external view. The methodology thenreturns to step 2002 and the methodology repeats.

FIG. 21 is an example methodology 2100 for interfacing with anindustrial automation system via a virtual control panel using awearable appliance. Initially, at 2102, a request for a virtual controlpanel is sent from a wearable appliance to a VR/AR presentation system,the virtual control panel associated with an automation system (ordevice) being viewed via the wearable device. In an example embodiment,the identification of the automation system or device can be obtainedbased on identification information read by the wearable apparatus froman optical code affixed to the automation system. In another exampleembodiment, the identity of the automation system can be inferred basedon the current location and orientation of the wearable appliance, asdetermined based on location and orientation data generated by thewearable appliance itself.

At 2104, user information is sent to the VR/AR presentation system fromthe wearable appliance. The user information can be provided to thewearable appliance by the wearer of the appliance, can be read fromstored user identification information on the wearable appliance, or canbe read from the wearer as biometric data (e.g., via retinal scans orother means). At 2106, a determination is made as to whether the userinformation sent at step 2104 identifies a user authorized to interfacewith the automation system. If the user information does not identify anauthorized user (NO at step 2106), the methodology ends. Alternatively,if the user information identifies an authorized user (YES at step2106), the methodology proceeds to step 2108, where the requestedvirtual control panel associated with the automation system is renderedon the wearable computer based on VR/AR presentation data received fromthe VR/AR presentation system.

At 2110, a communication channel is established between the wearablecomputer and an industrial network on which the automation system (ordevices thereof) reside. At 2112, data is exchanged with the automationsystem across the communication channel established at step 2110 viainteraction with the virtual control panel.

FIG. 22A is a first part of an example methodology 2200A for generatingaugmented reality presentations to guide plant personnel through theprocess of addressing a detected maintenance issue. Initially, at 2202,data is received from a set of industrial assets installed at anindustrial facility. At 2204, an augmented reality presentation isgenerated on a wearable appliance associated with a user, where theaugmented reality presentation is generated based on the industrial datareceived at step 2202 and one or more plant models that defineinformation about the industrial assets (e.g., locations and names ofthe assets, automation systems or production areas within the facilitythat are associated with the assets, etc.).

At 2206, a determination is made as to whether a maintenance issue isdetected based on a monitoring of the data received at step 2202. Themaintenance issue can be detected, for example, based on identificationof an alarm generated by an industrial controller, HMI, motor drive, orother industrial asset; a determination that a key performance indicatorof a controlled machine or process has fallen outside an allowabletolerance; detection of a possible maintenance concern based on analysisof audio or video data received from a video capture device installed atthe industrial facility; or other such detection scenarios.

If no maintenance issue is detected (NO at step 2206), the methodologyreturns to step 2202 and steps 2202-2206 repeat. Alternatively, if amaintenance issue is detected (YES at step 2206) the methodologyproceeds to step 2208, where a notification of the maintenance issue issent to the wearable appliance.

At 2210, a determination is made as to whether a workflow is availablefor the maintenance issue. The workflow can be a defined series of stepsknown to correct the detected issue, recorded as workflow data that canfacilitate generation of augmented reality presentations that guide theuser through the workflow. If no workflow is available for the detectedissue (NO at step 2210), the methodology ends. Alternatively, if aworkflow for the detected issue is available (YES at step 2210), themethodology proceeds to step 2212, where a determination is made as towhether there are uncompleted steps of the workflow remaining.

If uncompleted steps of the workflow still remain (YES at step 2212),the methodology proceeds to the second part 2200B illustrated in FIG.22B. At 2214, an augmented reality workflow presentation is rendered onthe wearable appliance. The augmented reality workflow is generatedbased on current location and orientation data from the wearableappliance, and graphically indicates instructions for performing acurrent step of the workflow. The augmented reality presentation cancomprise, for example, graphical indicators overlaid on the user's fieldof view indicating locations, machines, devices, or components to whichthe user's focus should be directed in order to complete the currentstep of the workflow, alphanumeric or vocal instructions explaining thecurrent step, video presentations demonstrating an action to beperformed by the user, or other such presentations. The augmentedreality presentation can continually update in accordance with theuser's changing location and orientation, as well as updating statusinformation for the industrial asset being addressed.

At 2216, a determination is made as to whether the current workflow stephas been completed. If the workflow step has not been completed (NO atstep 2216), the methodology returns to step 2214 and the augmentedreality presentation continues to be rendered. Alternatively, if thecurrent workflow step has been completed (YES at step 2216), themethodology returns to step 2212, where another determination is made asto whether steps of the workflow remain uncompleted. If additionalworkflow steps remain uncompleted (YES at step 2212), the methodologyagain executes steps 2214 and 2216 for the next step in the workflow.When there are no remaining workflow steps to be completed (NO at step2212), the methodology ends.

Embodiments, systems, and components described herein, as well asindustrial control systems and industrial automation environments inwhich various aspects set forth in the subject specification can becarried out, can include computer or network components such as servers,clients, programmable logic controllers (PLCs), automation controllers,communications modules, mobile computers, wireless components, controlcomponents and so forth which are capable of interacting across anetwork. Computers and servers include one or more processors—electronicintegrated circuits that perform logic operations employing electricsignals—configured to execute instructions stored in media such asrandom access memory (RAM), read only memory (ROM), a hard drives, aswell as removable memory devices, which can include memory sticks,memory cards, flash drives, external hard drives, and so on.

Similarly, the term PLC or automation controller as used herein caninclude functionality that can be shared across multiple components,systems, and/or networks. As an example, one or more PLCs or automationcontrollers can communicate and cooperate with various network devicesacross the network. This can include substantially any type of control,communications module, computer, Input/Output (I/O) device, sensor,actuator, instrumentation, and human machine interface (HMI) thatcommunicate via the network, which includes control, automation, and/orpublic networks. The PLC or automation controller can also communicateto and control various other devices such as standard or safety-ratedI/O modules including analog, digital, programmed/intelligent I/Omodules, other programmable controllers, communications modules,sensors, actuators, output devices, and the like.

The network can include public networks such as the internet, intranets,and automation networks such as control and information protocol (CIP)networks including DeviceNet, ControlNet, and Ethernet/IP. Othernetworks include Ethernet, DH/DH+, Remote I/O, Fieldbus, Modbus,Profibus, CAN, wireless networks, serial protocols, near fieldcommunication (NFC), Bluetooth, and so forth. In addition, the networkdevices can include various possibilities (hardware and/or softwarecomponents). These include components such as switches with virtuallocal area network (VLAN) capability, LANs, WANs, proxies, gateways,routers, firewalls, virtual private network (VPN) devices, servers,clients, computers, configuration tools, monitoring tools, and/or otherdevices.

In order to provide a context for the various aspects of the disclosedsubject matter, FIGS. 23 and 24 as well as the following discussion areintended to provide a brief, general description of a suitableenvironment in which the various aspects of the disclosed subject mattermay be implemented.

With reference to FIG. 23, an example environment 2310 for implementingvarious aspects of the aforementioned subject matter includes a computer2312. The computer 2312 includes a processing unit 2314, a system memory2316, and a system bus 2318. The system bus 2318 couples systemcomponents including, but not limited to, the system memory 2316 to theprocessing unit 2314. The processing unit 2314 can be any of variousavailable processors. Multi-core microprocessors and othermultiprocessor architectures also can be employed as the processing unit2314.

The system bus 2318 can be any of several types of bus structure(s)including the memory bus or memory controller, a peripheral bus orexternal bus, and/or a local bus using any variety of available busarchitectures including, but not limited to, 8-bit bus, IndustrialStandard Architecture (ISA), Micro-Channel Architecture (MSA), ExtendedISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB),Peripheral Component Interconnect (PCI), Universal Serial Bus (USB),Advanced Graphics Port (AGP), Personal Computer Memory CardInternational Association bus (PCMCIA), and Small Computer SystemsInterface (SCSI).

The system memory 2316 includes volatile memory 2320 and nonvolatilememory 2322. The basic input/output system (BIOS), containing the basicroutines to transfer information between elements within the computer2312, such as during start-up, is stored in nonvolatile memory 2322. Byway of illustration, and not limitation, nonvolatile memory 2322 caninclude read only memory (ROM), programmable ROM (PROM), electricallyprogrammable ROM (EPROM), electrically erasable PROM (EEPROM), or flashmemory. Volatile memory 2320 includes random access memory (RAM), whichacts as external cache memory. By way of illustration and notlimitation, RAM is available in many forms such as synchronous RAM(SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rateSDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), anddirect Rambus RAM (DRRAM).

Computer 2312 also includes removable/non-removable,volatile/nonvolatile computer storage media. FIG. 23 illustrates, forexample a disk storage 2324. Disk storage 2324 includes, but is notlimited to, devices like a magnetic disk drive, floppy disk drive, tapedrive, Jaz drive, Zip drive, LS-100 drive, flash memory card, or memorystick. In addition, disk storage 2324 can include storage mediaseparately or in combination with other storage media including, but notlimited to, an optical disk drive such as a compact disk ROM device(CD-ROM), CD recordable drive (CD-R Drive), CD rewritable drive (CD-RWDrive) or a digital versatile disk ROM drive (DVD-ROM). To facilitateconnection of the disk storage 2324 to the system bus 2318, a removableor non-removable interface is typically used such as interface 2326.

It is to be appreciated that FIG. 23 describes software that acts as anintermediary between users and the basic computer resources described insuitable operating environment 2310. Such software includes an operatingsystem 2328. Operating system 2328, which can be stored on disk storage2324, acts to control and allocate resources of the computer 2312.System applications 2330 take advantage of the management of resourcesby operating system 2328 through program modules 2332 and program data2334 stored either in system memory 2316 or on disk storage 2324. It isto be appreciated that one or more embodiments of the subject disclosurecan be implemented with various operating systems or combinations ofoperating systems.

A user enters commands or information into the computer 2312 throughinput device(s) 2336. Input devices 2336 include, but are not limitedto, a pointing device such as a mouse, trackball, stylus, touch pad,keyboard, microphone, joystick, game pad, satellite dish, scanner, TVtuner card, digital camera, digital video camera, web camera, and thelike. These and other input devices connect to the processing unit 2314through the system bus 2318 via interface port(s) 2338. Interfaceport(s) 2338 include, for example, a serial port, a parallel port, agame port, and a universal serial bus (USB). Output device(s) 2340 usesome of the same type of ports as input device(s) 2336. Thus, forexample, a USB port may be used to provide input to computer 2312, andto output information from computer 2312 to an output device 2340.Output adapters 2342 are provided to illustrate that there are someoutput devices 2340 like monitors, speakers, and printers, among otheroutput devices 2340, which require special adapters. The output adapters2342 include, by way of illustration and not limitation, video and soundcards that provide a means of connection between the output device 2340and the system bus 2318. It should be noted that other devices and/orsystems of devices provide both input and output capabilities such asremote computer(s) 2344.

Computer 2312 can operate in a networked environment using logicalconnections to one or more remote computers, such as remote computer(s)2344. The remote computer(s) 2344 can be a personal computer, a server,a router, a network PC, a workstation, a microprocessor based appliance,a peer device or other common network node and the like, and typicallyincludes many or all of the elements described relative to computer2312. For purposes of brevity, only a memory storage device 746 isillustrated with remote computer(s) 2344. Remote computer(s) 2344 islogically connected to computer 2312 through a network interface 2348and then physically connected via communication connection 2350. Networkinterface 2348 encompasses communication networks such as local-areanetworks (LAN) and wide-area networks (WAN). LAN technologies includeFiber Distributed Data Interface (FDDI), Copper Distributed DataInterface (CDDI), Ethernet/IEEE 802.3, Token Ring/IEEE 802.5 and thelike. WAN technologies include, but are not limited to, point-to-pointlinks, circuit switching networks like Integrated Services DigitalNetworks (ISDN) and variations thereon, packet switching networks, andDigital Subscriber Lines (DSL). Network interface 2348 can alsoencompass near field communication (NFC) or Bluetooth communication.

Communication connection(s) 2350 refers to the hardware/softwareemployed to connect the network interface 2348 to the system bus 2318.While communication connection 2350 is shown for illustrative clarityinside computer 2312, it can also be external to computer 2312. Thehardware/software necessary for connection to the network interface 2348includes, for exemplary purposes only, internal and externaltechnologies such as, modems including regular telephone grade modems,cable modems and DSL modems, ISDN adapters, and Ethernet cards.

FIG. 24 is a schematic block diagram of a sample computing environment2400 with which the disclosed subject matter can interact. The samplecomputing environment 2400 includes one or more client(s) 2402. Theclient(s) 2402 can be hardware and/or software (e.g., threads,processes, computing devices). The sample computing environment 2400also includes one or more server(s) 2404. The server(s) 2404 can also behardware and/or software (e.g., threads, processes, computing devices).The servers 2404 can house threads to perform transformations byemploying one or more embodiments as described herein, for example. Onepossible communication between a client 2402 and servers 2404 can be inthe form of a data packet adapted to be transmitted between two or morecomputer processes. The sample computing environment 2400 includes acommunication framework 2406 that can be employed to facilitatecommunications between the client(s) 2402 and the server(s) 2404. Theclient(s) 2402 are operably connected to one or more client datastore(s) 2408 that can be employed to store information local to theclient(s) 2402. Similarly, the server(s) 2404 are operably connected toone or more server data store(s) 810 that can be employed to storeinformation local to the servers 2404.

What has been described above includes examples of the subjectinnovation. It is, of course, not possible to describe every conceivablecombination of components or methodologies for purposes of describingthe disclosed subject matter, but one of ordinary skill in the art mayrecognize that many further combinations and permutations of the subjectinnovation are possible. Accordingly, the disclosed subject matter isintended to embrace all such alterations, modifications, and variationsthat fall within the spirit and scope of the appended claims.

In particular and in regard to the various functions performed by theabove described components, devices, circuits, systems and the like, theterms (including a reference to a “means”) used to describe suchcomponents are intended to correspond, unless otherwise indicated, toany component which performs the specified function of the describedcomponent (e.g., a functional equivalent), even though not structurallyequivalent to the disclosed structure, which performs the function inthe herein illustrated exemplary aspects of the disclosed subjectmatter. In this regard, it will also be recognized that the disclosedsubject matter includes a system as well as a computer-readable mediumhaving computer-executable instructions for performing the acts and/orevents of the various methods of the disclosed subject matter.

In addition, while a particular feature of the disclosed subject mattermay have been disclosed with respect to only one of severalimplementations, such feature may be combined with one or more otherfeatures of the other implementations as may be desired and advantageousfor any given or particular application. Furthermore, to the extent thatthe terms “includes,” and “including” and variants thereof are used ineither the detailed description or the claims, these terms are intendedto be inclusive in a manner similar to the term “comprising.”

In this application, the word “exemplary” is used to mean serving as anexample, instance, or illustration. Any aspect or design describedherein as “exemplary” is not necessarily to be construed as preferred oradvantageous over other aspects or designs. Rather, use of the wordexemplary is intended to present concepts in a concrete fashion.

Various aspects or features described herein may be implemented as amethod, apparatus, or article of manufacture using standard programmingand/or engineering techniques. The term “article of manufacture” as usedherein is intended to encompass a computer program accessible from anycomputer-readable device, carrier, or media. For example, computerreadable media can include but are not limited to magnetic storagedevices (e.g., hard disk, floppy disk, magnetic strips . . . ), opticaldisks [e.g., compact disk (CD), digital versatile disk (DVD) . . . ],smart cards, and flash memory devices (e.g., card, stick, key drive . .. ).

What is claimed is:
 1. A system, comprising: a processor, operativelycoupled to a memory, that executes executable components stored on thememory, the executable components comprising: a rendering componentconfigured to generate, on a first client device, an augmented realitypresentation of a portion of an industrial system located within a fieldof view of the first client device; and a monitoring componentconfigured to identify occurrence of a maintenance issue based on adetermination that industrial data from one or more industrial devicesof the industrial system satisfies a condition indicative of themaintenance issue; and a client interface component configured toreceive, from the first client device, a virtual note, an indication ofan industrial asset of the industrial system to which the virtual notepertains, and a viewing permission criterion for the virtual note,wherein the rendering component is further configured to, in response toidentification of the maintenance issue, retrieve workflow data defininga workflow for correcting the maintenance issue, and render, on thefirst client device, the workflow data as part of the augmented realitypresentation, and in response to determining that the industrial assetis being presented on a second client device, and that the second clientdevice satisfies the viewing permission criterion, render the virtualnote on the second client device.
 2. The system of claim 1, wherein theclient interface component is further configured to receive, from thesecond client device, another virtual note to be associated with theindustrial asset, and the rendering component is further configured to,in response to determining that the industrial asset is being presentedon the first client device, render the other virtual note on the firstclient device.
 3. The system of claim 1, wherein the viewing permissioncriterion is at least one of an indication of a user role permitted toview the virtual note or an identity of a specified person permitted toview the virtual note.
 4. The system of claim 1, wherein at least one ofthe first client device or the second client device is a wearablecomputer or a handheld device.
 5. The system of claim 1, wherein theclient interface component is configured to receive the indication ofthe industrial asset to which the virtual note pertains as at least oneof a gesture detected by the first client device or a verbal commandsubmitted to the first client device.
 6. The system of claim 1, whereinthe rendering component is configured to render the virtual note on thesecond client device as an augmented reality overlay on or near theindustrial asset.
 7. The system of claim 1, wherein the renderingcomponent is configured to render a virtual note icon on the secondclient device as an augmented reality overlay on or near the industrialasset, and to render the virtual note on the second client device inresponse to detecting a gesture or verbal command indicating selectionof the virtual note icon.
 8. The system of claim 1, wherein the clientinterface component is configured to receive the virtual note as spokentext submitted to the first client device.
 9. The system of claim 1,wherein the virtual note is an alphanumeric message.
 10. The system ofclaim 1, wherein the client interface component is further configured toreceive, from the first client device or a second client device, aspoken request for a current status of a specified industrial asset ofthe industrial system, and the rendering component is configured to, inresponse to receipt of the spoken request, retrieve industrial datapertaining to the current status of the specified industrial asset andrender, as an augmented reality overlay on the first client device orthe second client device, the industrial data.
 11. A method, comprising:generating, by a system comprising a processor, on a first clientdevice, an augmented reality presentation of one or more industrialassets located within a field of view of the first client device;identifying, by the system, occurrence of a maintenance issue based on adetermination that industrial data from the one or more industrialassets satisfies a condition indicative of the maintenance issue; inresponse to the identifying: retrieving, by the system, workflow datadefining a workflow for correcting the maintenance issue, and rendering,by the system, the workflow data on the first client device as part ofthe augmented reality presentation; receiving, by the system from thefirst client device, a virtual note, an indication of an industrialasset, of the one or more industrial assets, to which the virtual notepertains, and a viewing permission criterion for the virtual note; andin response to determining that the industrial asset is being renderedon a second client device, and that the second client device satisfiesthe viewing permission criterion, rendering, by the system, the virtualnote on the second client device.
 12. The method of claim 11, whereinthe receiving the viewing permission criterion comprises receiving atleast one of an indication of a user role permitted to view the virtualnote or an identity of a specified person permitted to view the virtualnote.
 13. The method of claim 11, wherein at least one of the firstclient device or the second client device is a wearable computer or ahandheld device.
 14. The method of claim 11, wherein the receiving theindication of the industrial asset to which the virtual note pertainscomprises receiving the indication as at least one of a gesture detectedby the first client device or a verbal command submitted to the firstclient device.
 15. The method of claim 11, wherein the rendering thevirtual note on the second client device comprises rendering the virtualnote as an augmented reality overlay positioned on or near theindustrial asset within a field of view of the second client device. 16.The method of claim 11, further comprising rendering, by the system, avirtual note icon as an augmented reality overlay on or near theindustrial asset within a field of view of the second client device,wherein the rendering the virtual note on the second client devicecomprises rendering the virtual note in response to detecting a gestureor verbal command indicating selection of the virtual note icon.
 17. Themethod of claim 11, wherein the receiving the virtual note comprisesreceiving the virtual note as spoken text submitted to the first clientdevice.
 18. The method of claim 10, further comprising: receiving, bythe system from the first client device or the second client device, aspoken request for a current status of a specified industrial asset ofthe industrial system, and in response to receipt of the spoken request:retrieving, by the system, industrial data pertaining to the currentstatus of the specified industrial asset, and rendering, by the systemas an augmented reality overlay on the first client device or the secondclient device, the industrial data.
 19. A non-transitorycomputer-readable medium having stored thereon instructions that, inresponse to execution, cause a system comprising a processor to performoperations, the operations comprising: rendering, on a first clientdevice, an augmented reality presentation of one or more industrialassets located within a field of view of the first client device;identifying an occurrence of a maintenance issue based on adetermination that industrial data from the one or more industrialassets satisfies a condition indicative of the maintenance issue; inresponse to the identifying: retrieving workflow data defining aworkflow for correcting the maintenance issue, and rendering theworkflow data on the first client device as part of the augmentedreality presentation; receiving, from the first client device, a virtualnote, an indication of an industrial asset, of the one or moreindustrial assets, to which the virtual note pertains, and a viewingpermission criterion for the virtual note; and in response todetermining that the industrial asset is being presented on a secondclient device, and that the second client device satisfies the viewingpermission criterion, rendering the virtual note on the second clientdevice.
 20. The non-transitory computer-readable medium of claim 19,wherein at least one of the first client device or the second clientdevice is a wearable computer or a handheld device.